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Executive Summary 

Daniel K. Aladjem 



Background and Study Purpose 

Tyack and Cuban (1995), in their seminal history of pub lie sehool reform, described the cycles of reform 
over the past century. What may well be the enduring legacy of the current wave of reform is the 
widespread prevalence of reform solutions developed outside the formal school setting — that is, reform s 
developed by proprietary firm s and nonprofit organizations. Some of these externally developed programs 
aim at single pedagogical challenges (such as reading instruction); others focus on social services (such as 
after school or summer school programs). The most prominent manifestation of this trend has been the 
advent of comprehensive school reform (CSR) models. CSR models aim to provide whole-school or 
comprehensive interventions that reengineer and redesign all aspects of school operations. Since the 
1990s, increasing attention and research efforts have focused on these models. Although research exists 
on the effectiveness of many individual models, very few large-scale studies of several CSR models have 
been conducted. ' 




The range of activity related to CSR is impressive. Thousands of schools have implemented CSR models 
over the past decade, using either self-developed models or externally developed models. This breadth of 
activity has spawned an almost equally wide variety of research into CSR. Despite the popularity of many 
CSR schools and models, as well as the research base on which these models rest, the work of the 
American Institutes for Research® (AIR®) in reviewing this literature (Herman et ah, 1999) demonstrated 
that, overall, research on the effects of different CSR models on student outcomes is less strong than had 
been assumed. Indeed, of the 24 models reviewed by AIR, only 3 had strong evidence of positive effects 
on student achievement. The recent meta-analysis by Borman et al. (2003) found much the same thing. 

By nature, researchers always think that more research is needed, but clearly, in this instance, such a 
recommendation for more research is at least an understatement. 



In September 2000, the U.S. Department of Education awarded to AIR a grant to conduct the National 
Longitudinal Evaluation of Comprehensive School Reform (NLECSR) with its partners, the University of 
South Florida and NORC at the University of Chicago. The grant was one of five awarded under The 
Comprehensive School Reform Research Grant Competition. The purpose of the program is to expand 
understanding of the full dimensions of school reform through rigorous investigation of the large-scale 
implementation of research-based CSR models as a strategy for increasing student achievement. 

Overview of the Study 

The NLECSR is a mixed-method study using quantitative and qualitative approaches to investigate 
patterns of key actors’ behavior, decision-making processes, and sets of outcomes. It employs a 
quasi-experimental design with matched treatment and comparison schools. It strives to determine the 



* Readers are direeted to the important work of Tushnet, Vemez and Goldhaber, Turnbull, and Slavin, all of whom 
are aetively eondueting major studies of CSR (Aladjem & Borman, in press). 
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effects of CSR models on student achievement in about 650 elementary and middle schools (grades 3-8), 
by identifying the components of CSR models that are most effective overall, as well as by describing the 
situations and populations for which specific CSR models are most effective. The mixed-method analysis 
examined the contextual supports that contribute to CSR model effectiveness. 

The 650 participating schools are located in 21 districts, primarily urban areas, across 17 states. NLECSR 
involved longitudinal surveys of district administrators (64), principals (650), and teachers (about 5,000) 
over the course of 3 years, as well as the collection of student record data (achievement and enrollment) 
for participating districts, schools, and classes. 

To complement national survey data, we also conducted qualitative research in 34 schools in five 
districts. These qualitative case studies contribute to our understanding of both the implementation and 
the effects of CSR models’ key components and overall model effectiveness. We observed classes to 
evaluate instruction; interviewed teachers and administrators about instruction, implementation, and their 
overall experiences with CSR efforts; and collected extant documents about reform, student achievement, 
and school demographics in each school. 

Three research questions (RQs), focusing on outcomes and implementation, drive the NLECSR. The three 
major questions are: 

♦ RQ 1 : How effective are specific externally developed, research-based CSR models in improving 
the achievement of all students? 

♦ RQ 2: How are model characteristics related to the success of model implementation and 
improvement in teaching and learning in specific types of settings and with specific types of 
students? 

♦ RQ 3 : What supporting conditions and strategies are necessary to effectively implement and 
sustain CSR models in schools and school districts? 

We examined CSR implementation in and impact on schools, and its progression through a series of 
phases from adoption to implementation to sustainability. In this report, we describe (a) the process 
whereby schools adopt CSR models and the implications of this process for implementation; (b) the 
relationship between the fidelity of model implementation and changes to the reform process within 
schools, that is, the relationship between implementation and important outcomes such as social capital 
and student achievement; and (c) the sustainability of reform within schools. We have organized our 
findings around the major life cycle phases and major outcomes of school-level reform. 

Initiating Reform 

The CSR process begins with the adoption process in schools. The literature is replete with references to 
the adoption process. A consistent theme in the literature, moreover, is the importance of the adoption 
process for later implementation. Three themes emerge as most salient: the degree to which the process is 
informative (providing adequate information to stakeholders) (Bodilly, 1998), inclusive (enabling 
stakeholders to have a voice in selecting a model) (Datnow et al., 2003), and legitimate (ensuring that 
stakeholders’ voices are uninhibited and unconstrained by external pressures) (Slavin & Madden, 1999). 
An equally consistent theme is the high frequency of the adoption process in schools not meeting these 
standards. 

In examining the adoption process, we looked at several specific subquestions: How prevalent are specific 
model adoption practices? Do adoption processes appear to vary by school and/or district context? How 
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informative, inclusive, and legitimate — in short the quality of the adoption process — were the adoption 
processes — in short, what was the quality of the adoption process — in the schools we studied? Lastly, to 
what extent is the nature and quality of the adoption process related to later implementation levels? 

Everyone Votes 

Overall, it appears that voting for a model has become an institutionalized practice: in 8 of the 21 districts 
in our sample, all of the principals reported that their faculty engaged in a voting process to select their 
model. In 10 other districts, over 70% of responding principals reported voting. In only one district did as 
few as 50% of schools report voting. 

To explain this high prevalence of voting, we looked both at district- and school-level factors. School size 
was the only factor that was significantly associated with voting. Large schools were more likely to 
engage in voting than small schools. However, survey data also indicate that large schools are less likely 
than small schools to reach a high percentage of faculty voting in favor of adoption. This is likely due to 
the relative ease of achieving consensus in a small school versus a large school and the relatively lower 
cost to large schools of voting instead of engaging in a consensus process. 

Voting Does Not Equal Open Faculty Participation 

The quality of the adoption process was categorized on an ordinal scale, which consisted of four 
categories ranging from closed to directed to guided to open adoption. Results revealed that the schools in 
the qualitative sample were distributed almost evenly across these four categories. The next section will 
provide a brief description of each of the types of adoption processes. 

Closed Adoption 

Schools with the lowest ratings were those in which the adoption process could be characterized as 
“closed.” In such schools, both the information-gathering and decision phases involved few staff, the 
former most often included a single individual or restricted set of staff, and the latter was either a forgone 
conclusion or teachers had no say whatsoever. Often (but not always), there tended to be a lingering sense 
of negativity with regard to the adoption process in such schools. 

Directed Adoption 

In these schools, there was one organization or individual who strongly encouraged the school to adopt a 
model, most often the principal. Quite often a vote to adopt the model did occur, but it was conducted 
primarily because most models require a formal vote process, and thus lacked legitimacy. 

Guided Adoption 

In schools with guided adoption, the information-gathering phase was often quite open and inclusive, and 
the decision phase legitimate, but at least one key aspect of the formalized adoption process was lacking. 
For example, a school may have fallen short with regard to the formats through which it provided 
information on the model choices; instead of visiting schools and inviting model developers to visit, 
stakeholders may have only reviewed videos and text materials. In such schools, teachers had a generally 
positive recollection of the adoption process. 

Open Adoption 

In these schools, there was clearly a proactive engagement on the part of many staff members with regard 
to model adoption. The information-gathering phase was extensive, often characterized by teachers 
visiting schools currently implementing models, making presentations to their colleagues, actively 
questioning model developers, and debating a range of model choices. The decision process in the schools 
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was legitimate, and teachers were allowed the opportunity for dissent. In short, all aspects of the process 
were “open,” including participation, model consideration, and feedback. 

Open Processes Do Not Guarantee Implementation 

While the preceding findings regarding the adoption process are of academic interest, the question of real 
policy importance, however, is to what extent is there an association between the openness of the 
adoption process and subsequent fidelity of implementation? Our survey data suggest that schools 
adopting CSR models through a voting process had greater school-level activity in decision making, a 
greater sense of clear and shared goals, higher levels of perceived usefulness of developer supports, and 
greater self-reported change in instruction in response to model-related professional development. Case 
study data suggest a relationship between the quality of the adoption process and the extent to which 
stakeholders understand the core components of their CSR model. The data taken together, however, also 
suggest a complex relationship. A closed adoption process did not preclude later buy-in and faithful 
implementation, and conversely, an open adoption process did not always overcome other implementation 
challenges. As McLaughlin (1990) noted in her followup to the RAND change agent studies, belief 
sometimes follows practice and “belief or commitment can follow mandated or coerced involvement at 
both the individual and system level.” 

Implementing Change 

By using quantitative and qualitative data, our analysis explored how CSR model implementation varies, 
identified what factors predict CSR model implementation, and illustrated how well school stakeholders 
understand CSR model implementation as well as the contextual factors that influence their 
understanding. 

Like most studies of CSR, this study applied both qualitative and quantitative measures of 
implementation. The qualitative analysis of implementation captured the process and comprehensiveness 
of implementation from the points of view of principals, teachers, and staff in a subsample of study 
schools. Our approach for the quantitative analysis was quite different. We measured implementation as 
fidelity, the extent to which the CSR model of interest was delivered to the intended recipients in the 
manner intended by the model’s developer (Aladjem, 2003). The approach that we have developed 
(Kurki, Aladjem, & Carter, 2005) is based on the idea that to measure the fidelity of implementation, we 
must measure what schools are doing and compare that with what CSR model developers consider to be 
“full” implementation. The challenge of measuring implementation was finding the difference between 
the positive, empirical reality of school life and the normative vision of CSR model developers. We 
compared survey data from schools to developers’ specifications for full implementation to calculate the 
level of implementation. For each treatment school’s matched comparison, we calculated an 
implementation value by comparing the comparison school to its matched treatment school’s CSR model. 
In this way we know how “model-like” practices were of our comparison schools as well as our treatment 
schools. 

CSR Schools Do Not Systematically Have Higher Levels of Implementation 
Than Their Matched Comparison Schools 

Surprisingly, we found comparable levels of implementation in CSR schools and their matched 
comparisons. The comparison schools in our sample exhibited high levels of implementation of 
model-like practices despite not formally implementing a CSR model. This may be due to the 
institutionalization of CSR practices (such as the 90-minute reading block pioneered by Success for All 
[SFA]), coordinated diffusion of model practices by district central offices, or organic mimetic activities 
of individual schools. 



.-\MHRICAN 
INSTITUTES 
FOR Research 



Executive Summary 



Implementation Varies by CSR Model 

Although the study found a few significant overall differences in implementation between CSR model 
schools and their matched comparison schools, vast differences existed across the various CSR models 
concerning the implementation of different components of CSR. In other words, the level of 
implementation was clearly related to the “bundle” of activities recommended or required by each CSR 
model provider as part of the model’s implementation. Thi s effect became especially clear over time, as 
the relationship between level of implementation and CSR model did not vary over time. Models like 
SFA continued to have high levels of implementation over time. Other CSR models tended to have lower 
levels of implementation concerning specific components of that CSR model. 

CSR Models Are Not Comprehensively Implemented 

CSR is supposed to be comprehensive, addressing several dimensions of school operations (governance, 
professional development, assessment, and instruction, among others). The results of the implementation 
analyses also revealed that only about one fifth of the schools in the study implemented CSR models 
comprehensively in 2002, and only about one tenth did so in 2004. Regardless of these findings, overall, 
schools implementing a CSR model were more likely to have high a comprehensiveness level than were 
comparison schools, both in 2002 and 2004. The low level of comprehensiveness could be attributed to 
selective implementation: schools may decide to implement only particular components of the CSR 
program, or schools may decide to implement different components sequentially, one after another. 

The Support of Key Actors Matters for Implementation 

Our results also show that principals’ instructional leadership, developers’ assistance, and teachers’ 
professional community were consistently and positively related to the level of implementation across 
many implementation indices. Similarly, English/language arts teachers were more likely than 
mathematics teachers to implement highly. School-level characteristics (such as percentage of students 
receiving ffee/reduced-price lunch, percentage of non-English-speaking students, and school size), 
although significantly related to some implementation indices, did not consistently predict level of 
implementation. These results illustrate the important role of agents (teachers, principals, model 
developers) in the implementation process, as opposed to the context considered by itself: CSR models 
can be successfully implemented in different environments if the relevant actors are engaged in the 
process of implementation. Another way of looking at this is to recognize the importance of school-level 
investment or ownership of the reform process. 

Finally, the research on implementation also led to the conclusion that of all the contextual variables 
considered, only one was significantly and consistently associated with the change in implementation: 
positive change in a principal’s instructional leadership increases the level of implementation. This 
finding again reinforces the importance of the principal’s leadership in CSR model implementation. 
Measures for teacher community and developers’ assistance were not found to predict change in 
implementation systematically across different implementation indices. 

Improving Student Achievement 

Typically the first question asked by policy makers and practitioners about CSR generally or about 
specific models is, “Does it work?” NEECSR examined the extent to which students in CSR schools 
improve on state mathematics and reading assessments relative to their peers in comparison schools. 
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CSR Works When Implemented 

The analysis found certain conditions under which CSR schools did experience higher academic 
achievement gains than their matched comparison schools. These conditions included high levels of 
implementation, implementing the model highly during years 3-5, and uniformity of high implementation 
across components. When these conditions were met, they led to significant improvement in academic 
outcomes, specifically in mathematics and reading. When these conditions were not met, there was no 
observed relationship between improvement in outcomes for CSR schools and their comparison group 
counterparts. In other words, the schools in both groups improved at the same rate. This relationship also 
was found to vary by model. 

The data show a statistically significant, positive relationship between level of fidelity of implementation 
and student achievement. With regard to specific models, we found that schools implementing SFA 
showed larger gains in student achievement than other models. Interestingly, the implementation analyses 
also showed that SFA schools overall implemented their model higher than both the other models and 
their comparison group schools. One possible interpretation here is that the prescriptive nature of SFA 
materials and instruction result in teachers being aware of the particular steps necessary to take in order to 
provide high-quality instruction. This finding is particularly important when considered alongside the data 
that demonstrate the interplay between implementation and achievement, particularly with regard to 
schools implementing Accelerated Schools Project, ATLAS Communities, and Co-nect. Our analyses 
demonstrate that were these models implemented as faithfully as SFA, the schools implementing those 
models would have shown greater improvement than the schools implementing SFA. The potential effect 
of these models may in fact be greater than for SFA. Our empirical observations, however, are that these 
models were not as highly implemented as SFA. 

Building Social Capital 

There are many ways to classify or categorize CSR models. Typically, CSR models tend to be grouped 
either as prescriptive or philosophical, that is, either as models that provide highly specific guidance to 
schools, often around classroom instruction or as models that provide a guiding philosophy and mission 
around which schools organize specific activities that are consistent with the model. Many models in the 
latter group emphasize activities such as fostering professional community among teachers and collective 
commitment, two dimensions considered indicators of teachers’ enhanced social capital. These models 
posit a view that long-term school improvement is dependent upon building social capital in schools. To 
test this theory, the study examined two specific questions. The first question asked, “To what extent is 
the growth in social capital in CSR schools associated with the implementation of externally developed 
CSR models?” The second question we posed was, “To what extent is social capital associated with 
student achievement among CSR model implementing schools?” 

CSR Builds Social Capital 

To examine social capital, we looked specifically at the extent to which CSR models developed a 
professional learning community and a collective commitment to school improvement and student 
achievement. We found that schools implementing either ATLAS Communities or Accelerated Schools 
Project exhibited greater improvement in social capital than their comparisons. This is not surprising, as 
both emphasize the importance of building professional communities with shared norms, values, and 
expectations at the core of their designs. 
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Social Capital Yields Achievement Gains 

Our analyses of the relationship between social capital and student achievement revealed no model 
effects, but did reveal a significant relationship between collective commitment and both reading and 
mathematics achievement. No similar relationship was found for professional learning community and 
mathematics achievement, though a positive, but not significant association, was found with reading 
achievement. 




Achieving Sustained Implementation 

Following “what works,” the other oft-asked question of policy makers and practitioners is how to sustain 
CSR. To examine sustained implementation, we looked at the extent to which schools implementing CSR 
in the 1 st year of our study maintained formal ties to their model developers over the course of our study 
and explanations for this sustained implementation. We then examined the change in fidelity of 
implementation that was associated with schools’ severing of formal relationships with CSR providers. 

Schools Drop CSR When Confronted by Multiple Risks 

We found that less than one third of the CSR schools in our sample ended their relationships with CSR 
model providers during the course of our study. The remainder sustained their relationships for at least 
3 years. To explore the reasons why one third of the sample severed ties with their developers, we 
examined 1 1 factors commonly cited in the literature as explanations for schools’ failure to sustain CSR. 
These factors are: low local school capacity, lack of political support, insufficient funding, lack of 
positive student outcomes, poor model-school fit/alignment, instability of school leadership, faculty 
turnover, lack of faculty commitment, lack of model specificity, lack of sustained professional 
development, and competing reforms. We found that none of the factors alone had a statistically 
significant relationship. Instead we found that these factors operate in combination, and that it is the 
accumulation of factors, not merely the existence of isolated events, that precipitates the cessation of 
formal ties with model providers. 

Implementation Continues Even After the Formal End of CSR 

When schools did sever formal ties to their model providers, we found a remarkably small, though 
statistically significant, decline in the fidelity of implementation. 
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Daniel K. Aladjem 
Andrea Boyle 



Introduction 

Tyack and Cuban (1995), in their seminal history of pub lie school reform, described the cycles of reform 
over the past century. What may well be the enduring legacy of the current wave of reform is the 
widespread prevalence of reform solutions developed outside the formal school setting — that is, reform s 
developed by proprietary firms and nonprofit organizations. Some of these externally developed programs 
aim at single pedagogical problems (such as reading instruction); others focus on social services (such as 
after-school or summer school programs). The most prominent manifestation of this trend has been 
visible in the advent of comprehensive school reform (CSR) models. CSR models aim to provide 
whole-school or comprehensive interventions that reengineer and redesign all aspects of school 
operations. Increasing attention has been paid to these models and to research on them. Although research 
exists on many individual models to demonstrate effectiveness, very few large-scale studies of multiple 
CSR models have been conducted. 

The range of activity related to CSR is impressive. Thousands of schools have implemented CSR models 
over the past decade, using either self-developed models or externally developed models. This breadth of 
activity has spawned an almost equally wide variety of research into CSR. Despite the popularity of many 
CSR schools and models, and despite the research base on which these models rest, the work of the 
American Institutes for Research® (AIR®) in reviewing this literature (Herman et ah, 1999) demonstrated 
that, overall, less strong research exists on the effects of different CSR models on student outcomes than 
had been assumed. Indeed, of the 24 models reviewed by AIR, only 3 had strong evidence of positive 
effects on student achievement. The recent meta-analysis by Borman and colleagues (2003) found much 
the same thing. By nature, researchers always think that more research is needed, but clearly, in this 
instance, such a recommendation for more research is at least an understatement. 

In September 2000, the U.S. Department of Education awarded to AIR a grant to conduct the National 
Longitudinal Evaluation of Comprehensive School Reform (NLECSR). The grant was one of six awarded 
under The Comprehensive School Reform Research Grant Competition. The purpose of the program is 
“to expand understanding of the full dimensions of school reform through rigorous investigation of the 
large-scale implementation of research-based comprehensive school reform models as a strategy for 
increasing student achievement.”^ 

After a brief overview of NLECSR, this chapter traces the relevant history of school reform and precursor 
movements that led to comprehensive school reform. 



2 “Office of Educational Research and Improvement; The Comprehensive School Reform Research Grant Competition; Notice Inviting Applications for New Awards for Fiscal 
Year (FY) 2000.” 65 Federal Register 11 (April 20, 2000) pp. 21284-21286. 
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Overview of the Study 

The NLECSR is a quantitative and qualitative study of behavior, deeisions, proeesses, and outeomes. It 
employs a quasi-experimental design with matched treatment and comparison schools. NLECSR seeks to 
determine the effects of CSR models on student achievement in about 650 elementary and middle schools 
(grades 3-8), identifying the components of CSR models that are most effective overall, as well as 
describing the situations and populations for which specific CSR models are most effective. We also note 
the contextual supports that contribute to CSR model effectiveness. The 649 participating schools are 
located in 21 districts, primarily urban areas, across 17 states. NLECSR involves longitudinal surveys of 
district administrators (64), principals (649), and teachers (about 5,000) over the course of 3 years, as well 
as the collection of extant student record data (achievement and enrollment) for those districts and 
schools. 

To complement national survey data, we also conducted qualitative research in 34 schools, 24 of which 
were implementing CSR models. These 34 schools are located in five districts. These qualitative case 
studies contribute to our understanding of both the implementation and the effects of CSR models’ key 
components and overall model effectiveness. We observed classes, interviewed teachers and 
administrators, and collected extant documents about reform, student achievement, and school 
demographics in each school. 

Three research questions, focusing on outcomes and implementation, drive the NLECSR. The three major 
questions are: 

♦ Research Question (RQ) 1 : How effective are specific externally developed, research-based CSR 
models in improving the achievement of all students? 

♦ RQ 2: How are model characteristics related to the success of model implementation and 
improvement in teaching and learning in specific types of settings and with specific types of 
students? 

♦ RQ 3 : What supporting conditions and strategies are necessary to effectively implement and 
sustain CSR models in schools and school districts? 

We examine CSR implementation in and impact on schools, and its progression, through a series of 
phases, from adoption to implementation to sustainability. We describe (1) the birth of school reform, that 
is, the process whereby schools adopt comprehensive school reform models; (2) the design of and 
findings from our survey measure of implementation fidelity; (3) changes to the reform process within 
schools, that is, the process of implementation and the contextual factors affecting implementation; and 
(4) the sustainability of reform within schools. Important indicators of “school health” are also examined 
along the way — the relationship between model implementation and student academic achievement as 
well as the relationship between implementation and social capital. Thus, we employ the metaphor of a 
life cycle of reform. We have organized our findings around the major life cycle phases and major 
outcomes of school-level reform. A brief review of the research on CSR will provide a context for 
reporting our findings. 

Towards a Theory of Comprehensive School Reform 

When the CSR movement rose to prominence in the late 1990s, it was regarded as an innovative, 
evidence-based, cutting-edge approach to school improvement. The principles behind this movement, 
though — notably the use of research-based school improvement models that are comprehensive and 
schoolwide in nature — have their roots in over 30 years’ worth of reform and research activity. In order to 
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fully understand the underlying questions revolving around the sehool reform literature, it is important to 
review the literature, and explore questions that have yet to be answered satisfaetorily 30 years later. The 
following seetions explore several preeursors of the present CSR movement, particularly the program 
evaluation and school effectiveness literature that informed the development and refinement of CSR 
practices as they exist today. 

Precursors 

Looking back nearly 40 years, three major initiatives presaged the development of CSR as we know it 
today: Project Follow Through, the effective schools movement, and, more recently, the work of New 
American Schools. The next sections outline the history of these efforts. 

Project Follow Through 

Initiatives to advance the use of externally developed school designs at the K-12 level can be traced back 
to Project Follow Through, a federally funded compensatory education program that field tested more 
than 20 different educational intervention models for underprivileged elementary school students. Aimed 
at bolstering the available evidence on effective instructional and child development practices (Flouse, 
1979; Kennedy, 1977; Egbert & England, 1992), Project Follow Through sponsored the implementation 
of theory-based school models in communities across the country and then assessed the outcomes of these 
strategies to determine which methods were most successful at improving the academic performance of 
disadvantaged students. The evaluation of Project Follow Through raised conceptual and methodological 
issues that held significant implications that resonate in the fields of program evaluation and educational 
research today. 

Authorized in 1967 as part of the Johnson administration’s War on Poverty, Project Follow Through grew 
out of a series of compensatory education efforts — practices designed to offset achievement-hindering 
factors related to poverty — that included Project Plead Start, a program offering educational, social, and 
health services to preschool aged children as a means of increasing their readiness to attend grade school. 
(Flaywood, 1982; Kennedy, 1977). When an evaluation of Plead Start indicated that any gains it produced 
had disappeared following children’s first year of kindergarten (Wolff and Stein, 1966), lawmakers 
established Project Follow Through to extend or “follow through” on Plead Start by serving its 
preschoolers as they attended kindergarten through grade 3 (St. Pierre, 1982; Egbert & England, 1992). 
Although Project Follow Through was initially envisioned to be a full-scale, nationwide social services 
program, funding cutbacks and a transfer of the program’s administrative authority to the Office of 
Education (OE) led to its redesign into a planned variation educational experiment that focused on a 
limited set of schools and districts (Flaney, 1977; Egbert & England, 1992; St. Pierre, 1982; McDaniels, 
1975; Flouse, 1979; Kennedy, 1977). 



Follow Through school sites were intended to select an intervention program from a set of conceptually 
oriented educational models and work with the developer or “sponsor” of that model to implement it in 
their local school communities (Egbert & England, 1992). Model sponsors, which ranged from 
universities to educational laboratories to commercial firms, agreed to support these implementations of 
their model by (1) providing technical assistance, guidance, and training to local participants and 
(2) monitoring and motivating their participants to ensure their implementation was comprehensive and 
coherent (Stebbins, St. Pierre, Proper, Anderson, & Cerva, 1977). At the height of the Project Follow 
Through program, some 22 different model sponsors were working with more than 170 sites nationwide, 
and as many as 30 additional community or parent-developed programs were also implemented over the 
course of the program. 
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Guidelines circulated by the Office of Education (OE) explained that Follow Through models should be 
designed to develop children’s ability to learn as well as address related issues including children’s self- 
esteem, motivation, autonomy, cultural background, and environmental support (Kennedy, 1977). 
Although models mostly shared this overall orientation, significant differences existed among the 
theories, practices, and strategies that each model embodied. The Project Follow Through evaluation 
(Stebbins et ah, 1977) categorized the models into three groups according to what evaluators determined 
to be the models’ main theoretical foci. Treatments dubbed “Basic Skills Models” favored a behavioristic 
approach rooted in the belief that all behaviors are learned and that underprivileged children fall behind 
developmentally and academically because they have not adequately been taught necessary social and 
academic skills. Examples of Basic Skills Models included the Direct Instruction model, the Behavior 
Analysis Model, and the Eanguage Development model. Cognitive or Conceptual Skills Models, the 
second category in the evaluation’s typology, were based on the theory that disadvantaged students’ 
academic performance scores resulted from their lack of cognitive experiences typically associated with 
children in their age group. These models, such as the High Scope Foundation’s Cognitively Oriented 
Curriculum, the Florida Parent Education Model, and the Tucson Early Education Model, tended to focus 
on using teacher-child interactions to build children’s conceptual skills. Finally, models that emphasized 
a psychodynamic approach or the theory that achievement is closely tied to children’s social and 
emotional growth were considered “Affective Skills Models;” the Bank Street College Model, Open 
Education Model, and Responsive Education Model were all considered part of this group (Adams & 
Engelmann, 1996). 

Developing an evaluation strategy for Project Follow Through proved challenging as ambiguity 
concerning the program’s multiple objectives clouded attempts to develop a set of universally accepted 
research questions. Evaluators eventually decided to focus their attention on which models or types of 
models produced the most gains in student outcomes (Haney, 1977; St. Pierre, 1982); however, the choice 
of specific student outcomes sparked considerable debate since models varied in their principal goals and 
intent. In the end, researchers settled on a standard battery of assessments used in the third grade, which 
included two cognitive or basic skills assessments and two affective assessments (Haney, 1977; 

Anderson, 1977).^ To determine the project’s impact on these test outcome measures, the national 
evaluation of Project Follow Through (Stebbins et ah, 1977) employed a quasi-experimental design by 
comparing a population of children who participated in Follow Through projects to a similar population 
of children who did not. Taking individual students as its level of analysis, the evaluation traced models’ 
assessment outcomes for four annual cohorts of students from the 1969-70 to the 1974-75 school year. 

The conclusions drawn from these analyses proved to be rather controversial. Results indicated that 
models emphasizing basic skills, particularly the Direct Instruction model, yielded greater gains both in 
basic skills and self-concept outcomes than models in other categories. Students from sites whose models 
focused on affective or cognitive outcomes (according to the evaluation’s model typology), tended to 
score even lower on basic skills assessments than they would have without having gone through the 
program. None of the model types appeared to be more successful at increasing students’ 



^ The Metropolitan Aehievement Test (MAT) and the Ravens Coloured Progressive Matriees were used as basie 
skills/eognitive assessments, while the two affective assessments were the Intellectual Achievement 
Responsibility Scale (lARS) and the Coopersmith Self-Esteem Inventory. 
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cognitive/conceptual scores (Stebbins et al., 1977)/ Coloring these findings, however, was the overriding 
issue of site variability; generally speaking, models’ effectiveness in improving student outcomes varied 
so significantly across different Follow Through sites that it overshadowed the variation across models, 
whose overall averages varied quite little/ Although the site-variability finding was consistent with the 
results of other multisite evaluations (Flaney, 1977), critics cautioned that the wide differences across sites 
had strong implications for the interpretation of the evaluation’s model-related findings. Anderson (1977) 
explains, “Had each model been judged by its results in one site only, and had that site been diabolically 
chosen, the ra nk order of model averages could have been almost reversed” (p. 221). 

The evaluation report drew a heavy response from critics who underscored numerous methodological 
issues that compromised the robustness of its results. Many of these problems arose when Project Follow 
Through’s programmatic requirements interfered with evaluators’ ability to establish rigorous 
experimental conditions (Elmore, 1975). One of the most basic limitations plaguing evaluators was that 
neither the Follow Through project sites nor the models used at those sites were randomly assigned. In 
keeping with the program’s legislative requirements, project sites were selected based on their high 
poverty concentrations, their capacity to administer health and social services (Kennedy, 1977), and their 
ability to meet Follow Through’s requirement to serve a mixture of urban/rural and public/non-public 
schools (Egbert & England, 1992). Moreover, in an attempt to maintain traditions of local educational 
control, participating sites were allowed to choose which model they would implement.® Concern about 
he lack of randomization also applied to the children among individual project sites as, in most cases, 
schools and students in the Follow Through communities were not randomly assigned to either receive 
the treatment or remain in the comparison group (Egbert & England, 1992; Haney, 1977).’ 

Critics also emphasized potentially bias-creating differences between Follow Through children and their 
comparison group peers. They pointed out, for instance, that some treatment and comparison populations 
were mismatched — at times “disastrously so” — on demographic characteristics such as race and family 
income (Gersten, 1984, p. 412).* Additionally, treatment and comparison group students’ initial academic 
preparedness may have differed systematically; the program requirement that at least half of a project’s 
treatment group be children who previously attended Head Start meant that treatment group students were 
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Other model-related findings suggested that certain models were more effective when used with particular 
populations of children; for example, some models — Direct Instruction, Parent Education, Behavior Analysis, 
Bank Street, and EDO — seemed to be more effective in sites with the most disadvantaged students, and two 
models were consistently more successful with Head Start children. However, evaluators warned that these 
findings should be treated with “less certainty and generality” than the other evaluation results (Haney, 1977, 
p. 244). 

Eleven of the thirteen models with three or more sites worth of data had at least one site with a positive average 
effect on basic skills outcomes, while all 13 of these models had at least one site that yielded a negative average 
effect (St. Pierre, 1982). 

The Office of Education urged communities to select the model that seemed the most appropriate for their 
particular population and circumstances; however, some community decision-makers indicated that models were 
also selected because they were perceived as less demanding or more consistent with what the community was 
already doing (Kennedy, 1977; Egbert & England, 1992). 

While the decision not to randomize students stemmed largely from Follow Through’s policy requirements (as the 
decision not to randomize sites had), randomizing students would also have raised various pragmatic challenges 
such as assigning and transporting children to different districts and/or schools (Egbert & England, 1992). 

Such mismatches might have resulted, for instance, from cases where all children participate in a Follow Through 
project, and the only available children to serve in the comparison group came from a different school district 
and/or a more advantaged socioeconomic background (McDaniels, 1975). 
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more likely to have received some preschool education (Haney, 1977).^ In conjunction with these 
baseline differences, students in the comparison groups were also tainted by contamination effects in 
cases where teachers who were not officially designated to implement a Follow Through treatment may 
have witnessed model practices performed in other schools or classrooms, and used these strategies on 
their own control group students (McDaniels, 1975; Haney, 1977). Furthermore, even if the comparison 
group children did not encounter the practices and services of a Follow Through model, they likely 
received some other form of compensatory education related to Title I or other programs (Haney, 1977; 

St. Pierre, 1982). 

Another source of contention involved how the evaluation defined and measured what the treatments 
(i.e. the models) actually were (Haney, 1977). Evaluators used the Basic Skills, Conceptual Skills, and 
Affective Skills model typology to determine whether any particular type of model was more likely to 
produce higher student outcomes, but some critics disagreed with these classifications, arguing that the 
category labels were misleading and that some models could reasonably be placed in more than one of the 
categories (House, Glass, McLean, & Walker, 1978). Further complicating attempts to define the model 
treatments was the fact that many model sponsors viewed Follow Through as a research and development 
endeavor, and thus refined and/or added components to their approaches over the course of the project, 
meaning that the treatments being evaluated in the early cohorts of Project Follow Through could have 
been markedly different from the treatments that were evaluated in later years (House, 1979; Haney, 

1977; Egbert & England, 1992). Furthermore, although researchers made early attempts to measure 
schools’ implementation of their models through classroom observations, in the end, evaluators had “no 
analytically useful record” (Stebbins et al., 1977, p. 166) regarding how extensively or faithfully sites had 
executed their model. Thus, treatments sites were measured only according to their model labels and 
classifications, which did not account for their actual dosage (Haney, 1977). 

Criticism generated in response to the Follow Through evaluation quickly prompted additional studies 
and commentary.'” By the time Follow Through ended in 1994, it had yielded a body of research 
addressing not only models’ outcomes but also numerous issues surrounding evaluation and educational 
program design (McDaniels, 1975; Elmore, 1975; Fullan, 1983; Haney, 1977). Although some critics 
questioned Project Follow Through’s ability to experimentally validate its models (Elmore, 1975), 
commentators acknowledged the program’s innovation in sponsoring the development of various 
theoretical and research-based school models, in promoting a reform strategy where model developers 
engaged with local stakeholders to provide implementation assistance, and in seeking to produce evidence 
of models’ effectiveness to inform school decision-making (Fullan, 1983; Kennedy, 1977). These 
activities later became a foundation for the comprehensive school reform movement, which called for the 
use of research-attested educational models as well as external, model-based implementation support. 

Effective Schools Research 

Further laying the groundwork for CSR was the Effective Schools movement, a widespread research 
campaign seeking to identify school-level factors associated with high student achievement, particularly 
among disadvantaged populations. For more than two decades, effective schools researchers investigated 
school conditions ranging from school culture to principal leadership to teacher attitudes and behaviors. 



” Evaluators did attempt to adjust for initial differences between the treatment and comparison populations, but 
some critics argued that these adjustments were based on unreliable criteria and had a tendency to disadvantage 
poorer students. 

See St. Pierre, 1982, and Egbert & England, 1992, for more about studies surrounding the Project Follow Through 
evaluation. 
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The resultant body of literature served as an empirieal basis for many eomprehensive sehool reform 
models (U.S. Department of Edueation, 2002). 

The Effeetive Sehools movement emerged during the early 1970s as edueational researehers aspired to 
eounter growing doubts regarding schools’ ability to make a difference in the lives of underprivileged 
youth. Such doubts were fueled in large part by a series of highly publicized studies (Coleman et ah, 

1966; Jencks et al., 1972) indicating that easily measurable differences among schools bore relatively 
little relationship to students’ academic achievement when compared to the effects of students’ family 
background and socioeconomic status. Some interpreted this overwhelming influence of family 
background to suggest that increased school improvement expenditures were unlikely to produce 
significant changes in student achievement since “research has found nothing that consistently and 
unambiguously makes a difference in student outcomes” (Averch et al., 1972, p. x). Concerned that such 
conclusions might lead to harmful changes in education policy and funding, researchers and activists set 
out to allay the growing pessimism about schools’ potential impact by finding evidence that schools can 
in fact demonstrate higher instructional effectiveness" and by identifying the characteristics that allowed 
them to do so (Marzano, 2000; Cuban, 1998; Rowan, Bossert, & Dwyer, 1983; Bickel, 1983; Jansen, 
1995; Mackenzie, 1983; Turkey & Smith, 1983). 

Commentators often attribute the popularity of this research endeavor to its underlying premise that if 
certain schools can effectively serve disadvantaged students, then other schools who adopt their 
characteristics and practices can achieve similar success (Miller, Cohen, & Sayre, 1985). This notion is at 
the heart of the effective schools movement’s ideology, which included the beliefs that (1) one can 
identify schools with the exceptional capacity to improve disadvantaged children’s achievement, (2) these 
schools demonstrate manipulable characteristics associated with that capacity, (3) such characteristics can 
serve as the basis for improving other less effective schools, and (4) the school is the appropriate level of 
improvement (Bickel, 1983). Once researchers identified characteristics related to schools’ success, they 
typically aggregated them into prescriptive lists or formulas for effective school environments. 

The literature on effective school practices employed several different study designs to locate factors 
correlated with school success. One design that gained early popularity was the outlier study, in which 
researchers identified and examined schools that were unusually effective or ineffective at determining 
student achievement given the family and socio-economic background of its student population. Based on 
these categorizations, researchers used surveys and/or case studies to uncover characteristics present in 
schools at each end of the student achievement spectrum (Turkey & Smith, 1983; Marzano, 2000; Jansen, 
1995). 

Related to the outlier study was the case study approach, which involved conducting in-depth 
examinations of particular schools or sets of schools that met certain achievement-related selection 
criteria. Some of these criteria included schools’ status as high- or low-performing (Weber, 1971; Ellis, 
1975; Venezky & Winfield, 1979) or as improving or declining (California State Department of 
Education, 1980; Brookover & Eezotte, 1979) in terms of their students’ reading, math, or overall 
achievement scores. The products of these case study investigations were often recipe-like lists of 
ingredients for highly effective schools, although in some cases, researchers used the case study approach 
to produce ethnographic portraits of what an “effective school” looked like (Jansen, 1995). 



" Although exact definitions of school effectiveness varied by study, most hinged upon the level or gain in 
achievement scores demonstrated by at-risk students (Turkey & Smith, 1983; Jansen, 1995; MacKenzie, 1983). 
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Evaluation studies of educational reform programs have also produced lists of school-level factors 
associated with effectiveness (Purkey & Smith, 1983). Program evaluations and implementation studies 
differed from the more descriptive outlier and case study analyses in that they examined interventions or 
changes in school behaviors (Marzano, 2000). As a result, these studies were interested in investigating 
what factors were linked to gains or losses in student achievement levels, and they featured school-level 
variables that were conceptually linked to the intervention program. Program evaluations tended to 
examine the school improvement process rather than just the effective school conditions these processes 
aimed to generate and thus shifted the research’s emphasis “from the cover of the book to its contents” 
(Mackenzie, 1983, p. 6). 

Over the course of the Effective Schools Movement, studies put forth numerous variables believed to 
contribute to schools’ instructional productivity. Many of these variables addressed schools’ overarching 
culture or climate (Halpin & Croft, 1963). For example, the research indicated that the faculty in effective 
schools shared common values and beliefs and agreed upon an approach to instruction (Rutter et al., 

1979; Phi Delta Kappa, 1980). School staff possessed collective goals that ultimately centered on 
academic achievement, particularly students’ acquisition of basic skills (Brookover & Eawrence, 1979; 
Phi Delta Kappa, 1980; Edmonds, 1982), and these goals to promote a learning-focused environment 
where staff “push” students to achieve (Edmonds, 1982; Stringfield & Teddlie, 1991). Effective schools’ 
teachers devoted more time to academic instruction (Brookover et al., 1978; Wyne & Stuck, 1982; 
Hallinger & Murphy, 1986) and ensured that students spend their time engaged and on task (Stringfield 
& Teddlie, 1991; Stallings, 1980; Mackenzie, 1983). They held high expectations for student achievement 
(Weber, 1971; Brookover et al., 1978; Edmonds, 1982; Rutter et al., 1979; Hallinger & Murphy, 1986) 
and believed students would continue their education at higher levels (Brookover & Eawrence, 1979). 
Moreover, teachers possessed a sense of efficacy (Brookover et al., 1978; Armor et al., 1976) and 
participated in instructional decision-making (Rutter et al., 1979). 

Most of these studies also discussed the principal’s role in fostering these favorable school conditions 
through strong leadership. Effective school principals reportedly were knowledgeable about school 
curricula and assertive in making instructional decisions (Weber, 1971; Edmonds, 1982; Brookover 
& Eawrence, 1979). They established a vision for the school (Weber, 1971) and were instrumental in 
facilitating the implementation of reform efforts (Phi Delta Kappa, 1980; McCarthy, Eazarus, & Canner, 
1980). Furthermore, principals actively advanced their faculty’s collective expertise by promoting 
professional development activities, recruiting highly skilled teachers and dismissing ineffective ones, 
evaluating teacher performance, and engineering professional learning communities that stimulate teacher 
collaboration and collegiality (Stringfield & Teddlie, 1991; Phi Delta Kappa, 1980; California State 
Department of Education, 1980; Venezky & Winfield, 1979; Sizemore, Brossard, & Harrigan, 1983; 
Rosenholtz, 1985). 



The intuitively appealing nature of the characteristics that many of these studies promoted helped 
Effective Schools research to gamer rather easy acceptance among education practitioners. However, 
amidst the profusion of school effectiveness studies during the 1970s and 1980s, a number of reviews and 
critiques of this literature (Purkey & Smith, 1983; Cuban, 1983; Clark et al., 1984; Ralph & Fennessey, 
1983; Rowan, Bossert, & Dwyer, 1983; Rutter, 1983) emerged from the educational research community. 
Although most commentators remained to some degree positive about the potential usefulness of the 
Effective Schools research, they expressed concerns regarding the generalizability of its findings and 
challenged the assumption that what works well in one school will also work well in others. Frequently, 
criticisms stemmed from limitations inherent in the study designs; some of the frequently cited 
methodological problems included small, uneven analysis samples; errors in the measures used to identify 
outliers; inappropriate comparison matching; a lack of control variables; observer bias; and reliance on 
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nonexperimental data (Purkey & Smith, 1983; Rowan, Bossert, & Dwyer, 1983; Ralph & Fennessey, 
1983). 

Measures of school effectiveness used by the studies further troubled critics. For instance, researchers 
often declared entire schools to be effective based on achievement data from one or two grade levels or 
from one or two subject areas (Rowan, Bossert, & Dwyer, 1983). Some critics also challenged the 
research’s almost sole focus on instructional outcomes as indicators of school success (Purkey & Smith, 
1983; Coe & Fitz-Gibbon, 1998). They argued that a more multidimensional view of school effectiveness 
that included administrative, social, and emotional variables would more fully capture schools’ capacity 
and could offer insight into how these other domains relate to achievement success (Rowan, Bossert, 

& Dwyer, 1983). 

Finally, commentators expressed concern about how this research lacked information regarding the 
process by which schools can acquire the characteristics that these studies promote. Purkey & Smith 
(1983) noted that the research’s implicit assumption that schools merely need to choose to incorporate 
effective school characteristics overlooked findings from the implementation literature which indicate that 
school change involves complicated adaptation processes. Related to the issue of how and whether 
schools could implement these variables was the problem of causal ordering; many of the recommended 
school conditions could have plausibly brought about or resulted from schools’ high achievement levels, 
and studies were often unable to determine this relationship (Rowan, Bossert, & Dwyer, 1983; Coe & 
Fitz-Gibbon, 1998). Another piece that critics argued was missing from this research was the relative size 
of the effects produced by various school conditions. An understanding of their relative effect sizes would 
hold strong implications for education practitioners who, faced with limited resources, must decide which 
characteristics are the most essential to promote (Rowan, Bossert, & Dwyer, 1983). To address these 
school-change issues, critics called for a greater focus on longitudinal analyses and investigations of 
“turnaround” schools (Rowan, Bossert, & Dwyer, 1983). 

Despite the research’s limitations, findings from these studies informed the development of CSR and 
other school improvement models and also provided a framework for many state and local school reform 
efforts (Bickel, 1983; Edmonds, 1982). The movement’s guiding principles — ^particularly its emphasis on 
raising student achievement, on improving educational equity for disadvantaged students, and on 
considering the entire school as the unit of reform — further shaped future educational policy and research 
(Cuban, 1998). Of particular significance for CSR was the concept of school culture or climate advanced 
by this literature. The notion that school characteristics interact to form a distinct school culture 
challenged the “view that schools are relatively static constructs of discrete variables” and instead implied 
that schools were complex systems in which the interplay of multiple factors contributed to student 
success (Purkey & Smith, 1983, p. 440). This conceptualization encouraged a move away from 
disconnected, piecemeal approaches to school improvement toward more comprehensive, schoolwide 
reform strategies (Pukey & Smith, 1983). 

The New American Schools Initiative 

The shift from fragmented to comprehensive, whole-school reform received a major boost in 1991 when 
the first Bush administration joined with business leaders to found the New American Schools 
Development Corporation (later, just New American Schools or NAS), a nonprofit organization intended 
to promote the creation and dissemination of comprehensive school reform models. Though privately 
funded, NAS originated as part of the President’s America 2000 education reform initiative, a response to 
heightening concerns throughout the 1980s and early 1990s that traditional schooling methods were 
preventing students from meeting world-class educational standards. NAS’s mission was to facilitate the 
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development of numerous “break-the-mold” approaehes to sehooling that would reinvent how sehools did 
business through the foeused use of research-based practices (Glennan, 1998; Mirel, 2001). 

The cornerstone of this objective was a belief spread by the Effective Schools literature that successful 
schools are characterized by a “unifying design” (Berends, Bodilly, & Kirby, 2002, p. 3) that permits 
efficient school functioning by means of a “coherent and mutually reinforcing set of effective approaches 
to teaching and learning for the entire school” (Ibid, p. xv). The idea was that these “unifying designs” 
could be replicated in compact reform models that schools would implement with the support and 
guidance of the model developers and NAS. Developers would provide hands-on training to school staff 
in the implementation of their unique reform models while NAS would provide political and financial 
backing behind the scenes. 

In the tradition of Project Follow Through, the NAS initiative involved a strong evaluative component. 
Shortly after its founding in 1991, NAS contracted with the RAND Corporation to provide research, 
analytic, and program evaluation support over the course of the organization’s work. RAND’s 
responsibilities were deliberately loosely defined in part to allow the flexibility needed to gauge what 
research would be necessary to evaluate an evolving organization and its still-evolving mission. RAND’s 
tasks included documenting how design teams’ approaches changed over time, determining schools’ level 
of model implementation, pinpointing factors that facilitated or inhibited implementation of CSR models, 
and evaluating whether adopting a model produced desired student outcomes. More broadly, RAND’s 
work involved articulating and assessing the validity of the theories underlying the NAS enterprise as 
well as its overall contribution to the field of school reform (Berends, Bodilly, & Kirby, 2002). 

Ultimately, RAND published more than a dozen official reports about NAS’s mission and its outcomes. 
Numerous other working papers and policy briefs were also released, and RAND’s body of work on the 
NAS project became one of the major repositories of research on comprehensive school reform during the 
mid- and late 1990s. 

NAS’s planned distribution of wholesale CSR models comprised four distinct phases: design selection 
(1992), development (1992-93), demonstration (1993-95), and scale-up (1995-98). In the selection or 
competition phase, the organization solicited proposals for the creation of these new models (or “unifying 
designs”) that could be delivered to schools as packaged reform strategies. In addition to the requirement 
that they be replicable across a large number of schools, models needed to be innovative, affordable, 
suitable to all student subgroups, and adaptable to local district and community contexts (Bodilly, 2001). 
Out of over 600 proposals, 1 1 CSR models were selected in the competition phase. They were ATLAS 
Communities; Audrey Cohen College; Bensenville (Illinois) New American Schools Project; Co-nect; 
Community Learning Centers; Expeditionary Leaming/Outward Bound (ELOB); Urban Learning 
Centers; Modem Red Schoolhouse; America’s Choice; Odyssey Project; and Roots and Wings. The year 
following the competition constituted NAS’s development phase during which the chosen model 
developers honed the programmatic content of their models and prepared them for implementation in 
schools. By the end of this second phase, NAS had abandoned its two locally developed designs, the 
Bensenville and the Odyssey Project, due in large part to community disagreements and funding issues, 
but also because their local orientation clashed with NAS’s increased focus on nationwide scale-up 
(Berends, Bodilly, & Kirby, 2002; Mirel, 2001). 

NAS’s third stage was a demonstration phase in which the nine remaining teams piloted their designs in 
two or more schools and made further adjustments to prepare their models for duplication in multiple 
settings. The demonstration phase also saw the publication of RAND’s first major reports. NAS had 
requested that RAND conduct formative assessments of the design teams’ work during the initiative’s 
early phases to keep stakeholders apprised of their progress and to identify pertinent lessons learned that 
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would inform the future scale-up process (Bodilly et al., 1996; Bodilly et al., 1995). In response, RAND 
performed comparative case studies of the design teams to investigate such matters as how essential 
characteristics of the models and design teams affected their progress toward NAS’s goals, whether the 
teams had effective strategies for implementation assistance, and what obstacles needed to be addressed 
to achieve successful scale-up (Bodilly et al., 1996). 

With respect to the models themselves, these reports indicated that (1) the nature of the designs largely 
varied from one another and (2) the designs changed significantly over the course of the initial three 
phases. Such changes were attributed to planned development from design teams, school-level adaptation 
of the designs to meet the needs of teachers, and other modifications in response to school, district, and/or 
state policies (Bodilly et al., 1996; Bodilly, 2001). These observations of model characteristics had 
implications for schools’ ability to fully implement their models as the nature or type of model was 
associated with the variation in schools’ implementation. Other factors related to design teams’ success in 
demonstrating full implementation of their models included the teams’ preparedness at the start of the 
initiative as well as the implementation strategy they employed in schools (Bodilly et al., 1996). 

An examination of the design teams’ implementation strategies during the demonstration phase revealed 
the importance of design-based assistance (Bodilly, 2001). To underscore the significance of this 
developer support, future RAND reports referred to what were formerly called “design teams” as 
“design-based assistance teams” (Bodilly, 2001). RAND’s investigation into which aspects of design 
teams’ assistance strategy helped improve implementation progress in schools produced five major 
themes: (1) an inherent vision that guides school staff, (2) an implementation strategy that “guides the 
sequencing of tasks” and implementation assistance, (3) progress assessments and associated mechanisms 
for adjusting efforts based on assessment outcomes, (4) resources sufficient to implement change over a 
relatively short period, and (5) ongoing professional development in content related to model 
implementation. 

In addition to this developer assistance, it became clear to NAS that implementing reform models 
entailing substantial changes to nearly every aspect of the school enterprise required strong support from 
district administrators (Bodilly et al., 1996; Bodilly, 2001). Comprehensive school reform simply 
involved too many drastic changes for individual schools to manage successfully without assistance and a 
wide delegation of authority from local administrators. In light of their observations in that respect, NAS 
chose to adopt a “concentration strategy” for its nationwide scale-up, rather than promulgating CSR 
models indiscriminately and risking failure in districts uncommitted to these reform efforts. Instead, NAS 
would partner with a limited set of school districts with an eye toward implementing CSR in a high 
percentage of schools (Berends, Bodilly, & Kirby, 2002). RAND theorized that the presence of effective 
designs and assistance strategies to implement them would exert subtle pressure on school districts to 
delegate authority to schools to make the models work. 

The nationwide scale-up, the fourth and final phase of the initiative, officially lasted from 1 995 until the 
academic year ending in 1998 and represented the true launch of NAS’s plans to expand the number of 
CSR models in low-performing districts. Seven design teams'^ were selected to transmit their model to a 
wider sample of schools. In accordance with its concentration strategy, NAS focused its scale-up in 10 
under-performing districts,'^ each of which committed to involve at least 30% of its schools in the 



These were ATLAS Communities; Audrey Cohen College; Co-neet; ELOB; Modem Red Schoolhouse; Ameriea’s 
Choiee; and Roots and Wings. 

These distriets ineluded Cineinnati, OH; Dade County, FL; Memphis, TN; Philadelphia, PA; Pittsburgh, PA; San 
Antonio, TX; San Diego, CA; and smaller distriets in Washington State. 
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initiative. The individual schools chosen to implement NAS designs typically scored below the district 
average on state assessments and tended to serve high-poverty and often high minority student 
populations (Berends, 1999). 

To assess and document the NAS scale-up process, three major research projects were undertaken that 
formed the core of RAND’s investigation into NAS and CSR. These were a longitudinal study of 
implementation and performance (Berends et ah, 2001; Kirby, Berends, & Naftel, 2001; Berends, Bodilly, 
& Kirby, 2002), case studies of implementing schools (Bodilly, 1998; Bodilly & Berends, 1999; Chun, 
Gill, & Heilbrunn, 2001), and a detailed classroom study in San Antonio, TX (Berends et ah, 2002). In 
general, RAND focused on two variables throughout its analysis of NAS: model implementation and 
student achievement as a result of model exposure. RAND primarily explored the extent to which models 
were adopted in the schools and districts affiliated with NAS and what contextual factors impacted the 
extent of implementation. It also investigated whether or not student achievement gains were realized as a 
result of model adoption. 

RAND developed two implementation indices with which to measure schools’ implementation of the 
NAS designs: one index measured the broad changes believed by NAS to be co mm on to each reforming 
school, and a second index measured design-specific changes, taking into account variations between 
schools and districts flowing from different emphases and approaches used by each model. The 
longitudinal analysis of NAS scale-up schools showed modest overall gains in implementation between 
1997 and 1999. Schools’ implementation levels tended to increase during the initial 4 years after adopting 
a model but decreased in the years that followed. Implementation levels varied largely by school, school 
district, and NAS design. Moreover, the within-school variance of schools’ implementation levels 
exceeded the between-school variance and increased over time, indicating that models were not 
implemented schoolwide. Factors infiuencing implementation included the NAS designs themselves, 
model-specific technical assistance, schools’ model selection process, district context, and schools’ 
capacity and context. (Berends, 2000; Berends et al., 2001; Kirby, Berends, & Naftel, 2001; Berends, 
Bodilly, & Kirby, 2002). 

Regarding student achievement outcomes, an analysis of 1 63 design-implementing schools revealed that 
of the schools, half made gains in mathematics in comparison to other schools in the district, while 47% 
showed gains in reading. Researchers found evidence indicating that strong principal leadership was 
associated with higher student achievement; however, the instructional practices prescribed by the NAS 
designs were not significantly related to student achievement in comparison to other school- and 
classroom-level conditions. Schools’ generally low implementation levels might explain this lack of an 
effect (Berends et al., 2002; Berends et al., 2001; Kirby, Berends, & Naftel, 2001; Berends, Bodilly, 
Kirby, 2002). 

Reflecting on the NAS initiative’s legacy for future educational reform, RAND evaluators acknowledged 
the organization’s role in fostering the development and widespread use of numerous comprehensive 
school reform models. They further recognized NAS’s emphasis on sponsoring research and evaluation 
work related to this enterprise. The literature resulting from the NAS initiative not only provided evidence 
of NAS models’ effectiveness, it also yielded several lessons about the reform process that could be 
applied to later efforts. For instance, RAND’s evaluation reports focused attention on the district- and 
school-level context for reform. They questioned the feasibility of implementing complex, nontraditional 
designs in what were often highly regulated, low-capacity schools and stressed the importance of 
addressing school and jurisdictional conditions that could inhibit model implementation. Moreover, to 
avoid complications related to the reforms themselves, RAND’s reports cautioned model developers 
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against prematurely taking their models to scale before the reform design and implementation strategy 
were clearly defined and tested. 

Furthermore, the NAS evaluations underscored the need for schools to receive design-oriented technical 
assistance to facilitate their implementation of external interventions (Berends, Bodilly, & Kirby, 2002). 
Although the use of design-based assistance already had a long tradition in the field of education reform, 
NAS’s innovation resided in the fact that it “deliberately set out to develop a variety of design-based 
assistance organizations with which schools can affiliate . . . [and] invested not only in creating the 
designs themselves but also in developing organizations and their strategies for engaging and assisting 
schools to implement their designs” (Glennan, 1998, p. 23). 

Title I and the Comprehensive School Reform Demonstration Program 

The federal government’s largest role in advancing comprehensive approaches to school reform involved 
a series of Title I policies that encouraged and financed the implementation of whole school reform 
strategies. Under the 1994 reauthorization of the Elementary and Secondary Education Act (ESEA), the 
federal Title I program, which offered schools resources to assist disadvantaged students, included 
provisions allowing schools in which at least 50% of the student population qualified for free or 
reduced-price lunch to use their Title I funds to support the implementation of schoolwide reforms. These 
provisions arose amidst concerns that the pull-out approach employed by Title Ts predecessor, chapter 1, 
was distracting at-risk children from their school’s curriculum by removing them from the classroom and 
providing supplemental instruction that was not always aligned with the regular school day curriculum. 
The schoolwide provision of Title I extended its services to all students, promoting a consistent and 
proven approach to instruction across the entire school population. Since comprehensive school reform 
models were deemed an acceptable use of these funds, it is not surprising that CSR soon experienced its 
most rapid growth among Title I Schoolwide schools (Slavin & Fashola, 1998). 

As CSR models’ popularity grew in response to the NAS initiative and the Title I Schoolwide program, 
federal lawmakers endeavored to bolster this reform strategy by alleviating the potentially prohibitive cost 
of CSR models and developer assistance. In 1997, Congress established the Comprehensive School 
Reform Demonstration Program (CSRD),''* which provided schools and districts with grants of at least 
$50,000 that were renewable for up to 3 years. Intended to serve as “start-up” funds to initiate schools’ 
CSR work, these grants were administered by SEAs through a competitive process that focused on CSR 
proposals’ ability to meet nine federally required components: (1) the use of proven, scientifically based 
methods and strategies; (2) a comprehensive design with aligned components; (3) a professional 
development strategy; (4) measurable goals and benchmarks; (5) support within the school; (6) parental 
and community involvement; (7) external technical support and assistance; (8) annual evaluations; and 
(9) coordination of resources. Four years later, this funding stream was renamed the Comprehensive 
School Reform Program and was incorporated into the No Child Eeft Behind (NCEB) Act (2001) as Title 
I, Part F. NCEB mandated two additional components: (1) schools were required to locate resources to 
sustain their CSR work after the 3-year funding cycle, and (2) proposed CSR models needed to have 
strong evidence of their ability to improve student achievement. 

Additional CSR Research 

While RAND’s analyses of the NAS initiative provided one of the few large-scale evaluations of multiple 
CSR models, numerous other CSR studies soon followed as the reform’s growing popularity increased 
demands for research-based support strategies and evidence of CSR models’ effectiveness. For the most 



CSRD grants also became known as “Obey-Porter” funding after the appropriations bill amendment that 
authorized the program. 
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part, these CSR studies were more foeused or limited in scope. Some investigated specific district or state 
CSR efforts such as the Memphis Restructuring Initiative (Ross et ah, 2001; Smith et ah, 1998; 

Stringfield & Ross, 1997), the Comer Schools and Families Initiative in Detroit (Millsap et ah, 2000), and 
the court-mandated use of CSR in New Jersey’s Abbott districts (Erlichson & Goertz, 2001). Other 
researchers probed CSR’s implications for special school settings such as rural locales (Carlson & 
Buttram, 2004) and multicultural communities (Datnow et al., 2003) or for special student populations 
such as English language learners (Hamann, Zuliani, & Hudak, 2001; Berman et al., 2000; Menken, 

2000) and students with disabilities (Jurich & Duffy, 2003; Purnell & Claycomb, 2001). 

A large proportion of CSR research focused on the outcomes of a particular CSR model. These studies 
included evaluations conducted by model developers themselves such as Slavin and his colleagues’ 
publications on Success for All (SEA) (Slavin et al., 1996; Slavin & Madden, 1999) and Comer and his 
associates’ reporting on the School Development Program (Haynes, Emmons, & Woodruff, 1998; Comer, 
1998; Haynes, Comer, & Hamilton-Eee, 1988). They also included several independent evaluations of 
CSR models like the Accelerated Schools Project (Bloom et al., 2001), Co-Nect (Ross & Eowther, 2003), 
the School Development Program (Cook, Hunt, & Murphy, 1999), America’s Choice (Supovitz & May, 
2004), Core Knowledge (Datnow, Borman, & Stringfield, 2000), Coalition of Essential Schools (Muncey 
& McQuillan, 1993), and Talent Development (Kemple, Herlihy, & Smith, 2005). 

CSR Model Reviews and Research Syntheses 

As CSR studies proliferated and federal policy began to focus increasing attention on researched-based 
and scientifically proven practices, several reviews emerged to guide practitioners’ use of the burgeoning 
CSR research. Slavin and Fashola (1998) sought to discourage schools and districts from making reform 
decisions according to “the pendulum swings of current fashion” (viii) and urged the education 
community to instead select strategies based on their record of effectiveness and replicability. To 
facilitate such a decision-making process, they summarized the available research on numerous 
schoolwide reform programs and described strategies for supporting the school change process. 

In 1999, AIR released An Educator’s Guide to Schoolwide Reform (Herman et al., 1999), which 
examined the existing evidence surrounding 24 popular reform models’ effects on student achievement. 
Using a consistent set of standards, this study rated and compared models according to their reliable 
evidence of effectiveness. The study demonstrated that, overall, less strong research on the effects of 
different CSR models on student outcomes existed than had been assumed. Of the 24 models AIR 
reviewed, only 3 had strong evidence of positive effects on student achievement: Direct Instruction, High 
Schools that Work, and SFA. 

The recent meta-analysis by Borman and colleagues (2003) confirmed this weakness in the research base, 
concluding that although CSR did appear to have promising effects on student outcomes, some of the 
available research may have overstated the size of these effects. They expressed particular concern over 
CSR evaluations that employed less rigorous study designs as well as evaluations conducted by model 
developers and argued that independent, experimental or quasi-experimental studies were likely to 
generate more objective, reliable results. 

OERI Research Grants 

As noted above, in September 2000, the U.S. Department of Education awarded to AIR a grant to conduct 
the NEECSR. The grant was one of six awarded under The Comprehensive School Reform Research 
Grant Competition. The following section describes the approach we took to this evaluation. 
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Conceptual Framework 

The study design (deseribed at length in ehapter 2) follows the research questions originally promulgated 
in the Federal Register notice announcing the grant program. To address these questions, we developed 
the conceptual framework depicted in Figure 1.1. 

We have characterized this framework as a life cycle model. Through it, we examine the implementation 
and impact of comprehensive school reform from the time schools first initiate and adopt CSR, through 
implementation, to student achievement to long-term sustainability. Along the way, we investigate the roles 
of states and districts in supporting CSR, as well as the influence social capital has on student achievement. 

Figure 1.1. Conceptual framework 




Organization of This Report 

The purpose of this chapter has been to provide a quick introduction to the study and important 
background for the work we have conducted. The remainder of this report will describe the methods used 
(chapter 2), the state and district context for CSR (chapter 3), the adoption process (chapter 4), 
implementation of CSR (chapter 5), the role of social capital (chapter 6), student achievement gains 
(chapter 7), and the sustainability of CSR (chapter 8). We conclude with some brief policy implications 
and recommendations. 
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Introduction 

This chapter provides an overview of the study design. Details of particular analyses are provided in each 
chapter with additional detail provided in the Technical Appendixes. 

We designed the study to address the very broad and all-encompassing questions posed by the federal 
government (described in chapter 1). We began with certain assumptions about the best way to address 
such a broad range of questions in a single study. These assumptions were that the study should: 

♦ Employ a quasi-experimental design; 

♦ Use both qualitative (case study) and quantitative (surveys and extant data) methods; 

♦ Examine multiple, externally developed CSR models (not just one or two models); and 

♦ Account for the role we assumed local school districts play to the greatest extent possible. 

To study multiple models, we selected eight prominent CSR models that cover the spectrum of 
approaches to comprehensive school reform, from highly scripted to more philosophical. In alphabetical 
order, the models were: Accelerated Schools Project, ATEAS Communities, Co-nect, Expeditionary 
Eearning/Outward Bound (EEOB), Modem Red Schoolhouse, Success for All (SFA)/Roots and Wings, 
Turning Points, and Urban Teaming Centers. Our expectation was that these models were so concentrated 
in urban areas that we would be able to select a sample of schools implementing these models in a sample 
of the largest districts. To do this, we requested and received rosters of schools implementing each model 
from each model provider. We discovered that these models were not as concentrated in large urban 
districts as we and the model providers had thought. Consequently, we expanded the scope of our district 
sample and ended up with unequal numbers of schools per model in each district. We maintained a 
sample of schools nested within districts because of our belief and the body of literature suggesting the 
importance of accounting for district variation. In addition to the eight models selected, we also created a 
ninth category consisting of schools not implementing any of our eight models, but receiving federal CSR 
funds in our sample of districts. 

To answer our research questions, we drew upon three sources of data: survey data, case study data, and 
extant achievement data. We describe the use of each next. 

Survey 

We collected survey data over 3 years, from 2001 to 2004, from principals and district officials using 
self-administered, paper-and-pencil questionnaires the first year and telephone surveys in the last 2 years. 
One principal per school and 3-4 district officials per district were sampled. In addition, teachers 
completed self-administered paper-and-pencil questionnaire in the 1st and 3rd years of data collection. 
Approximately 10 teachers per school were sampled. AIR subcontracted with NORC at the University of 
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Chicago to collect the survey data. AIR developed all survey instruments. AIR and NORC jointly 
designed and selected the school and student sample. 

District and Schooi Sampiing 

AIR and NORC defined the universe file of possible treatment and control schools for NLECSR using the 
U.S. Department of Education’s 1999-2000 school year Common Core of Data (CCD) file. AIR and 
NORC identified potential treatment schools by obtaining lists of member schools from each of the eight 
CSR model providers. The models selected included Accelerated Schools Project, ATEAS Communities, 
Co-nect, Expeditionary Eearning-Outward Bound, Modem Red Schoolhouse, Success for All, Turning 
Points, and Urban Teaming Centers. These were selected because they were among the most prominent 
models nationally. They also represent a range of approaches to comprehensive school reform. AIR and 
NORC proposed and implemented a “ninth stratum” classification to include CSRD schools 
implementing a model other than one of the eight original models. 

District Selection and the Master Database 

NORC and AIR jointly decided to restrict the sample to schools from the districts with the largest 
numbers of CSR schools. The primary reason for this decision was our interest in urban education. We 
also wanted to keep treatment-control comparisons within districts so as to yield more valid comparisons 
than cross-district comparisons would allow. Additionally, larger districts improved the odds of finding 
well-matched treatment and control schools within a district. Several studies sponsored by the U.S. 
Department of Education (ED) focused on evaluations of comprehensive school reform, and in order to 
avoid multistudy participation at the school level, ED solicited MPR (a research organization) and NORC 
to maintain a “master database” (referred to as MDB) of ED-funded grant-based studies as well as several 
contract studies being conducted under contract from ED in the fall of 200 1 . The MDB aided the selection 
of districts and flagged any potential collisions, that is, a school targeted for more than one study. After 
“collisions” were identified, all grantees and contractors involved negotiated district and school 
participation based on their respective sampling frameworks, the importance of school factors/models and 
geography among other criteria. 

School Selection 

The goal of the matching procedure was to pair each CSR (“treatmenf ’) school with a non-CSR 
(“control”) school within the same school district, one which resembled the CSR school as closely as 
possible in terms of factors related to the main dependent variable of the NEECSR project, student 
achievement gains. NORC relied on measures of student socioeconomic status and race/ethnicity. Within 
districts, the grade span offered by each school constrained the pool of controls eligible to be matched 
with each treatment. As a result of grouping schools by grade span, not all control schools would have a 
chance to be matched with all treatments. Treatment and control schools were matched using cluster 
sampling in terms of a “Euclidean distance” measure. 

In each selected district, NORC and AIR selected the treatment schools with certainty based on the CSR 
model adopted by the school. The eight focal CSR models constituted eight strata in the treatment sample; 
in addition, schools with a federal CSR grant and a CSR model other than one of the focal models form a 
ninth stratum in the treatment sample. The control population includes all schools other than those drawn 
into the treatment population, with the exception of schools considered out of scope (00 S) (typically for 
the reason of covering the wrong grades). Only schools in the 48 contiguous states and schools with at 
least two grades in the 3-8 range were eligible for NEECSR. 



2i 



.-\MHRICAN 
INSTITUTES 
FOR Research 




Chapter 2: Study Design 



Gaining Cooperation 

Initially, a formal and lengthy phase of gaining cooperation was not explicitly detailed in the NLECSR 
plan. As the study commenced, NORC planned to send a notification letter to the district and school 
offices introducing officials to the NLECSR and then have minimal phone follow-up by field staff to 
answer questions or address concerns. During the initial stages it became clear that gaining cooperation at 
both the district and school level involved more of an official approval process and considerably more 
field effort and prompting than expected, as will be outlined below. Before the start of gaining 
cooperation, NORC IT staff assigned NORC IDs (SUIDs) to all districts and schools. These IDs were 
unrelated to any IDs assigned by any other organization or government agency in order to maintain 
confidentiality. Throughout the gaining cooperation process and all three data collection rounds, NORC 
maintained a toll-free number and e-mail account for study participants to contact with questions. 

District Cooperation 

In late June 2001, after sending notification letters to state officials, NORC sent via FedEx an advance 
package to each of the 22 districts selected for the NLECSR. This package included a personalized 
advance letter and a brochure giving a brief overview of the NLECSR. The Superintendent and Title I 
Coordinator in each district office received this district cooperation package. In late June 200 1 , trained 
NORC field managers who followed up on the advance letter by phone, answered any questions from the 
district officials to whom they spoke, and inquired about any prerequisites to district approval. 

One step involved at the district level not detailed in the study proposal was the requirement of a 
“research clearance package” at some district offices. A combination of generic clearance package 
submissions, complex and personalized packages, and verbal discussions with various staff at the district 
level resulted in 21 out of 22 districts granting permission to AIR and NORC to approach schools in their 
districts. After districts gave their verbal consent or approval of the submitted research clearance package, 
field staff obtained a signed consent form from each district. The length of the entire process of Gaining 
Cooperation at the district level varied greatly. The last district came on board in January 2002, 
overlapping with the gaining cooperation phase of previously released districts’ schools. 

Endorsing Agencies 

During the district gaining cooperation period, NORC staff also worked on gaining the endorsement of 
several education-related agencies. NORC displayed the names of the four endorsing agencies on the 
back of the brochure used for the school gaining cooperation phase of the study. The endorsement letters 
served also as a part of an approach for difficult or refusal districts and schools. Field staff shared the 
letters with such schools and districts as substantiation of the credibility of the NLECSR. 

School Cooperation 

Since the 2 1 districts agreed to participate in a period of time spanning from early summer 200 1 through 
the beginning of 2002, NORC released the schools from these districts for gaining cooperation in waves. 
The first mailing occurred in September of 2001 and the last in January 2002. After the initial mailing, 
NORC released schools as soon as districts granted permission, whether verbal or written, and released 
entire districts at a time. Districts that employed a structure incorporating regional districts were 
exceptions. In these districts, NORC mailed to all treatments in the consenting regions and only controls 
for treatments already released as regional districts granted consent. Across the nine model strata and 
control schools, 1,084 schools were released into the gaining cooperation phase and 649 came on board 
and subsequently entered the first data collection round. During gaining cooperation, 26 schools sampled 
for the NLECSR were found to be OOS. 
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NORC mailed a letter, broehure, roster memorandum, agreement letter form, and business reply envelope 
(BRE) as part of the initial sehool paekage addressed to the school principal. A few districts, as part of the 
consent stipulations, also requested that NORC send schools a copy of the signed district consent letter 
and/or approved research clearance package. The “roster memo” included in the package stated the 
request from schools of a school roster of teaching staff with subject and grade indicated along with full 
teacher names. In addition, it specified the types of teachers to be included: grades 3-8, teaching either 
mathematics or English/language arts (or both). 

From the national pool of NORC field interviewers, NORC central office staff trained 30 field 
interviewers with experience in school or similar institutional studies and good refusal conversion skills. 
Four field managers supervised the field interviewers and worked with central office staff on approaches 
and any difficult issues that arose. Training of field staff occurred in waves to coincide with the release of 
districts assigned to the interviewers. Field staff commenced school phone prompting in September of 
2001 and continued their efforts through January 2002. Much of the gaining cooperation work was done 
over the phone, though several schools received in-person visits from NORC field staff. Field 
interviewers worked with central office staff to formulate approaches and focus efforts and addressed 
principals’ questions and concerns. Central office staff, through the monitoring of the case management 
system and weekly calls with field managers, produced analyses of schools prioritizing schools 
implementing certain models and certain vital treatment-control pairs. 

In addition to prompting for written consent and rosters, field staff also requested information such as 
school calendars and the name of an appointed NFECSR coordinator in the school to utilize as a primary 
contact during data collection. With the help of the NORC case management system and field staff, 

NORC singled out schools for special refusal conversion letters (late 2001) and prompting letters (January 
and February 2002) or in-person visits (January 2002) to obtain agreement letters and teacher rosters. 
While the majority of school cooperation ended in late January and staff consolidation occurred, NORC 
staff continued to prompt for agreement letters and rosters. 

Rostering and Teacher Sampling 

Rostering 

As mentioned, NORC collected teacher rosters from schools agreeing to participate in the NFECSR. 
Schools sent in their hard copy rosters via USPS (using NORC-supplied BREs), e-mail, or fax. All were 
receipted in the NORC production center and subsequently “preedited,” and data were entered into a 
roster data-entry system. Preediting by trained NORC staff was required so that only teachers in scope per 
the sampling rules of the study were data-entered and sampled. While NORC requested an up-to-date 
roster of teaching staff identifying grade and subject area(s) taught, many “rosters” included all school 
staff (e.g., administrators and maintenance workers) as well as OOS teachers (e.g., kindergarten-level 
teachers and special education teachers). Trained NORC staff crossed out all staff other than teaching 
staff in grades 3-8 who taught either mathematics or English or both subjects. IDs were also assigned at 
this time. These NORC-assigned SUIDs were unrelated to any school-assigned staff ID rosters may have 
contained. 

Teacher Sampling 

School respondents selected for participation in the NFECSR, not including principals, included all 
general classroom teaching staff (excluding special education and English as a second language staff) 
who fit the parameters of (1) teaching grades 3, 5, 6, and/or 8 and (2) either taught “all subjects” (or 
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NORC assigned one of two versions of the questionnaire (either English/language arts or mathematics) to 
that teacher. Where the subject area of a teacher was perceived to be “general or all subjects,” NORC 
implemented a random assignment process to determine the questionnaire version for these cases. 

For those schools where NORC field staff received a verbal consent but did not receive a roster in a 
timely manner, NORC implemented an imputation process. A description of this process is available in 
the data collection section below. 

Round I Data Collection 

The data collection phase of the first round of the study began in January 2002. This phase overlapped 
with the gaining cooperation phase for a few weeks as NORC worked to secure final approval to 
approach the selected schools in the few remaining districts. The NLECSR encountered significant 
changes in the mode of implementation from the proposal stage of the study to when the study actually 
entered the data collection stage in early 2002. Originally, the project proposal envisioned the following 
assumptions that changed once concrete preparation began: little to no field follow-up or financial 
incentives required, and the employment of telephone interviewing as the primary mode of data 
collection. Ultimately, the project evolved to include a self-administered paper and pencil questionnaire, 
monetary incentives for participating teachers and school coordinators, coupled with extensive field 
follow-up, both by phone and in person. These changes were for the most part due to the fact that NORC 
would not be able to contact teachers directly via phone. Teachers often do not have direct phone access 
at school and schools would not likely give out teachers’ home phone numbers. 

Questionnaire Development 

AIR and NORC finalized all three of the primary NEECSR questionnaires and arranged for printing. The 
three types of questionnaires for school respondents were principal questionnaire, teacher questionnaire — 
English/language arts version, and teacher questionnaire — mathematics version. A district official 
questionnaire was also developed and printed; however the quantity was significantly lower than for the 
school respondents. 

School Mailouts 

NORC released a given school district’s schools into data collection based on the percentage of schools 
from which NORC had received and data-entered a complete teaching staff roster. The earliest were 
mailed in late winter 2002 and the last were mailed in late spring 2002. NORC sent packages including 
the principal questionnaire and teacher questionnaires to the designated school coordinator or principal 
(when a school coordinator had not been designated during gaining cooperation) at each school. The 
NEECSR opted to send all mailings to schools by overnight FedEx delivery to ensure quick receipt of 
materials and easy online or phone tracking of packages. NORC production center staff placed the 
following materials in each individual presentation envelope: a cover letter describing the study and 
participatory incentives, an individually labeled questionnaire, an NEECSR brochure, and a prepaid first 
class BRE. All presentation envelopes and questionnaires were labeled with the respondent’s name, grade 
level, questionnaire version, teacher or principal identification number, and school identification number. 
Finally, a school coordinator letter outlining directions for distribution, a school coordinator incentive 
form, and a BRE was placed on top of the prepared principal and teacher presentation envelopes prior to 
being placed in the FedEx Pak/box. 

Imputation Cases 

During the gaining cooperation phase of NEECSR, a number of verbally consenting schools submitted 
only part of the necessary paperwork. Despite a number of follow-up attempts to retrieve the missing 
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paperwork throughout the final weeks of the gaining eooperation effort, several schools did not submit 
rosters listing active teaching staff in a timely manner. Using data from the CCD, NORC sampling staff 
imputed the number of teachers in these schools and sampled using this data. The two imputation 
mailings occurred in early and mid May 2002 and included approximately 60 schools. Assignment of 
questionnaires and IDs to teachers was facilitated by field interviewers, either in person (the majority) or 
by phone. 

District Official Mailout 

In spring 2002, NORC mailed questionnaires to officials at each of the 21 participating school districts. 
Previously, AIR had decided to sample three respondents in each of these districts, selecting staff working 
in the following capacities: superintendent, director of curriculum and instruction, and Title I coordinator. 
To the degree that it was possible, NORC staff found the appropriate persons occupying those positions in 
each of the participating districts. In a few districts, due to the nature of overlapping job descriptions and 
titles, NORC and AIR selected an additional district official respondent. District respondents received a 
cover letter, properly labeled questionnaire, brochure, and BRE for the questionnaire’s return. 

Field Contacting 

During the course of the data collection phase for Round I, over 30 field interviewers, located in the 
17 states in which districts were sampled for the NLECSR, contributed their efforts to secure the 
maximum benefits and results for the NEECSR. For the most part, NORC field interviewers were the first 
personal contact with many of the teachers for the NEECSR, although all principals involved had 
previous contact with NORC field staff by phone during the gaining cooperation phase. Interviewers were 
trained by phone in waves as schools entered the data collection phase of the study, from March 2002 
through April 2002. NEECSR field interviewers proved successful in accomplishing the following tasks, 
either by phone or via in-person visits: 

♦ Following up with those schools (after they had received their questionnaire and prompting 
mailings) deemed by AIR or the NORC central office to be priority schools and/or having low 
response rates; 

♦ Working with gatekeepers and/or a predesignated NEECSR school coordinator to prompt for 
questionnaires in person for teacher and principal respondents; 

♦ Answering any questions that arose; 

♦ Obtaining a letter of consent from schools (when applicable); 

♦ Securing teacher rosters (when applicable, for schools who consented to participate in the 
NEECSR but did not return a roster to NORC during the gaining cooperation phase; these schools 
were not released into data collection until a roster was received or an imputation process was 
completed.); and 

♦ Coordinating between the schools, principals or teachers, and the NORC field managers. 

For four districts where NEECSR staff was unable to staff field interviewers because of geographical 
constraints to visit participating schools in person, NORC staffed field interviewers to follow up with 
teachers and principals by phone. Fortunately, the field interviewers responsible for contact with these 
particular school districts proved to be highly effective in refusal conversion and other important 
strategies. Furthermore, the response rates for these districts are also indicative of the high level of school 
to field interviewer and field manager contacts through this medium. 
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Also as a part of the field phone follow-up efforts, toward the very end of the data collection phase, a 
senior field manager was charged with contacting all the outstanding District Administrator Questionnaire 
respondents. Once the data collection efforts began winding down, the field manager assigned to this task 
commenced the phone follow-up efforts with the district offices. To accommodate the district officials’ 
busy schedules, the field manager completed a number of the questionnaires over the phone with the 
respondents. Other officials completed the questionnaires and returned them in the provided BREs. The 
field manager’s efforts resulted in the completion of nearly 100% of the district official questionnaires. 

Unlike many studies where field interviewers attempt to complete a face-to-face interview with a 
respondent, the main field interviewer task of this phase of the NLECSR, as mentioned before, was to 
gather and/or prompt for completed questionnaires for the study. Prior to these in-person visits however, 
NORC production center staff mailed prompting letters directly to all teachers that had already (it was 
hoped) received questionnaires in the originally mailed batches to the school coordinators. 

Prompting Remail 

In late May, toward the end of data collection efforts in the field, AIR and NORC decided to send 
individual questionnaires, via FedEx packages, to 2,600 outstanding teachers and principals in all but 
three districts, which had already closed for the school year. This effort, executed by the NEECSR central 
office staff in conjunction with NORC production center staff, served as a final effort to bring response 
rates to satisfactory levels prior to the end-of-year closure of the remainder of the school districts in the 
study. The success of this effort appeared to be in that the packages were sent via FedEx directly to 
teachers at the schools with no intermediary serving as a possible impediment or obstacle to the delivery 
of the packages to the teachers. The teachers were able to complete the questionnaires and mail them back 
individually using materials included in the FedEx package. 

Incentive Distribution 

AIR coordinated the accounting and distribution of incentive checks to teachers and school coordinators. 
In this study, teachers who submitted their completed questionnaire received a $20 incentive check. 
Principals and district officials who returned their completed questionnaire did not receive an incentive. 
Also at each school, the school nominated or assigned a school coordinator (typically the principal, but 
sometimes another administrator, school official, or teacher at the school) to help coordinate the data 
collection efforts at that school. He/she was also eligible for the $20 incentive, even if he/she served 
concurrently as both principal and NEECSR school coordinator. Incentives were mailed to teachers on a 
biweekly basis at their school addresses in individual envelopes. Only on an ad hoc and individual basis 
were checks mailed to nonschool addresses. As an additional “thank you,” NORC also ordered pens with 
the NEECSR name and logo for distribution to school teachers, administrators, and any other 
“gatekeepers” that project staff may have encountered during the course of the study. 

Data Entry 

NORC central office staff and technology staff collaborated to create the computer-assisted data entry 
(CADE) system to capture data from the questionnaires collected during data collection. The CADE 
instrument included skip patterns found in the questionnaire; thus the production center staff followed the 
logic of the hardcopy questionnaire on the screen as they entered the data. Data entry occurred once 
questionnaires were received and receipted, and continued throughout data collection. 

Case Review 

In order to ensure a clean sample, NORC staff reviewed questionnaires and field staff reported 
information gained from schools concerning OOS cases, whether incomplete or complete. Although the 
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roster protocol was developed to create a sample of only in-scope cases, OOS cases gained entry into the 
sample due to incomplete roster information and staff changes and turnover. NORC reviewed various 
issues with AIR to develop a policy for handling cases and maintained a spreadsheet with all issues and 
delivered this to AIR for use in analysis. In addition to OOS cases, NORC also reviewed and addressed 
issues of teachers receiving a questionnaire for a subject that they did not teach. Where possible, these 
respondents were asked to complete the “correct” version of the questionnaire. These instances were 
generally the result of incomplete or inaccurate roster information or a change in assignment. 

Data Review and Delivery 

Throughout data collection, NORC conducted quality control measures to ensure the highest quality data. 
NORC staff reviewed frequencies from the beginning of data entry. In addition, separate data entry staff 
verified 20% of all cases entered and reconciliation of cases occurred, making sure that all cases expected 
were data entered. Prior to any deliveries to AIR, NORC developed and implemented data cleaning 
specifications. In June 2002, NORC delivered an interim dataset, codebook, and memo to AIR containing 
300 cases. The final delivery, including the final dataset and supplementary files, occurred in fall of 2002. 
Table 2.1 reports response rates for Round 1. 

Round II Data Collection 

Round II of data collection for the NLECSR required modifications to the previous round’s data 
collection methods. The chief differences between the two rounds creating the need for modifications 
were respondent sample scope and size. As the NLECSR data collection plan calls for teacher data 
collection in Rounds I and III, Round II only retains principals and district officials in the sample. Based 
on the sample scope for Round II, NORC and AIR decided to switch from paper-and-pencil interviewing 
(PAPI) to computer-assisted telephone interviewing (CATI) given that principal and district officials 
maintain access to telephones during the school day. This induced a change from using field managers 
and field interviewers in a management and prompting capacity to utilizing telephone interviewers (TIs) 
to complete phone interviews from the telephone shop at NORC’s Chicago production center facility. 
These changes required alteration to Round Es instrument, case management, training schedule, 
reporting, and overall data collection management. 

Instrument Development 

NORC began preparations for the principal and district official data collection instruments in late 2002 
and early 2003. While incorporating edits to the previous round’s instruments provided by AIR, NORC 
translated the instruments to a CATI mode and developed the accompanying TNMS (telephone number 
management system). By utilizing the computer-assisted data entry (CADE) instruments already in place, 
NORC preserved previously created instrument skips and formatting. Finally, because a set of questions 
in the Round II principal instrument functioned with the use of specific Round I data, NORC IT staff 
loaded the needed data into the system. All systems were thoroughly tested prior to release. 

Mallouts 

One week prior to the interviewer training, an advance mailing to 714 respondents occurred from the 
production center by FedEx. NORC and AIR determined that the principal sample include all 649 
principals from the cooperating schools, regardless of individual principals’ participation in the prior 
round. In addition, NORC also sent the advance mailing to all 65 district officials selected during the 
prior round. Both packages included an advance letter reintroducing the NLECSR and asked respondents 
for their participation in a phone interview. Follow-up packages were faxed on an individual basis to 
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Telephone Interviewing Staff 

Due to the mode change, PAPI to CATI, NORC chose to utilize its Chicago telephone interviewing center 
and staff for the NLECSR, rather than the field staff used in Round I. NLECSR’s production center 
coordinator recruited and trained veteran NORC staff with experience in studies dealing with institutions 
(e.g. hospitals) as well as well-qualified newly hired staff in early March 2003. The in-person training of 
eight TIs included information on the study as well as directed practice with the CATI instrument. 
Interviewers who passed the thorough certification process for the NEECSR began to call sample 
members in the days following the training. They followed up on the advance mailing, answered 
questions, and attempted to complete interviews with the principals and district officials. 

Prompting Mailing 

In early May 2003, the Telephone Center supervisors created a list of pending principal and district 
official cases; consequently the production center mailed about 260 refusal conversion and prompting 
letters by FedEx. 

School Closings and Position Elimination 

During data collection, NORC Telephone Center staff discovered three OOS principal cases and one OOS 
district official case. The principal OOS cases resulted from schools closing prior to the 2002-03 school 
year. The OOS district official case occurred in a small district where the position was eliminated. The 
viable district official able to replace the respondent in the eliminated position was already a respondent 
under another ID in the NEECSR. Therefore, NORC was unable to replace the respondent in the study. 

Data Review and Delivery 

NORC staff thoroughly reviewed data gathered by the TIs, and 20% of cases were verified by supervisory 
staff NORC delivered Round II data on June 20, 2003, after careful review and preparation. A list of 
dispositions for all cases was included in the delivery files. NORC delivered 51 district cases and 571 
principal cases. Table 2.2 reports response rates for Round II. 
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* Three schools closed for the 2002-03 school year and thus the principal cases for those have been made OOS. 



** One district official position was eliminated; we had already interviewed the suggested respondent at this district. 

Round III Data Collection 

As in the previous round, Round III of data colleetion for NLECSR required modifications to the previous 
rounds’ data collection methods. Like Round II, principals and district officials were asked to complete 
telephone interviews, thus CATIs and TIs at NORC’s production center were utilized once again. Unlike 
Round II however, a teacher sample was again asked to participate in the study. The necessary rostering 
process implemented in Round III, as will be discussed in further detail below, went through some 
modifications based on lessons learned in Round I. Because of the difficulty of contacting teachers by 
phone, the study maintained the teacher survey mode from the original round, paper-and-pencil 
interviews, and utilized NORC field managers for prompting at schools. Since teachers were participating 
in this round, NORC and AIR also resurrected the Round I incentive process for teachers and 
coordinators. There were slight modifications to the survey instruments as specified by the client. To 
accommodate the use of both the production center and field staff, alteration to previous rounds’ case 
management, training schedule, reporting, and overall data collection management was necessary. 

Instrument Development 

NORC began preparations for the principal, district official, and teacher data collection instruments in fall 
2003. The modes remained the same as the preceding rounds so there was no instrument SurveyCraft 
translation required in Round III for the three instruments. As in the previous round, AIR provided edits 
to the previous rounds’ instruments and NORC implemented the changes and additions to the preexisting 
electronic instruments. NORC also reinstituted the TNMS system used in Round II. AIR utilized the copy 
of the Round I teacher PAPI questionnaire to make changes for the paper questionnaire which would be 
mailed to the teachers in the NLECSR sample. NORC staff in the central office and production center 
tested all instruments and approved them for data collection. Linally, because a set of questions in the 
Round III principal instrument functioned with the use of specific Round II data, NORC IT staff loaded 
the necessary data into the system. 

Teacher Rosters and Sampling 

In late 2003 through early 2004, NORC collected teacher rosters over the phone in order to create the 
teacher sample for Round III. Due to the varying quality and format of the hard copy rosters received 
during the first round, a different approach of collecting teacher rosters was used in this round. By 
collecting the teacher roster information over the phone, NORC field staff could immediately clarify and 
verify any information given, rather than calling back after receiving a hard copy where teachers’ 
eligibility for the study, grades, and subjects might be missing or unclear. Lollowing their November 
2003 training, field managers verified, corrected, and updated the original roster obtained in Round I. 
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The final step of the rostering proeess was the assignment of an eight-digit SUID to new teaehers. As in 
the first round, NORC randomly assigned questionnaire versions where teaehers were deemed a 
“generalist” (teaehing all subjects including math and English) in the roster file. Starting from the last 
in-scope teacher from the original round, new teachers were assigned the subsequent version, alternating 
versions for each teacher that followed. 

Mailouts 

NORC sent an advance mailing for Round III to principals and district officials in late January 2004. This 
mailing was sent approximately a week prior to the interviewer training and included a letter reminding 
respondents of the study, asking for their participation in a phone interview. In all, 696 respondents 
received either a principal (631) or district official (65) letter. In mid-February, the production center 
mailed bundles of teacher questionnaires to each of the 63 1 schools participating in Round III. Each 
school package was made up of individual teacher packets. The 5,635 teachers received a cover letter, 
either a mathematics or English/language arts questionnaire with a label (including the teacher’s SUID), 
and a prepaid business reply envelope printed with the production center facility’s address all within an 
envelope marked with the teacher’s information (including name and SUID). The individual teacher 
packets were packaged together with a cover letter instructing the school contact concerning the 
questionnaire distribution. The materials in this package mentioned the $20 incentive for teachers who 
completed the questionnaire and the school contacts who served as the study coordinators. 

Field Contacting 

Over the course of data collection, several training sessions via phone were held to accommodate field 
staff brought onto the project mid-phase. The original training was held in February 2004. Field managers 
called to prompt for teacher questionnaires and answer any questions. Also, some schools received 
in-person prompting visits by field interviewers, managed by the field managers. Field interviewers 
provided extra teacher questionnaires, responded to questions, and picked up completed teacher 
questionnaires where possible. Calls to schools began following the teacher questionnaire mailing. 

Telephone Interviewing Staff 

In Round III, NORC again collected principal and district official data via telephone interviews. To 
conduct the interviews, NORC utilized its Chicago telephone interviewing center facilities and staff. In 
early February 2004, a 1-day in-person training session took place at the production center’s training 
facility, involving nine TIs. As in the prior round, the interviewers were trained to be able to answer 
questions as well as complete the CATI with principals and district officials. TIs who were certified 
following their training began to call sample members to follow up on the advance mailing, answer 
questions, and attempt to complete the survey over the phone. 

Prompting Mailings 

In mid-April, NORC conducted a mass prompting mailing via FedEx where new questionnaires were sent 
individually to teachers from whom NORC had not yet received a questionnaire and who had not been 
deemed OOS. This was in addition to regular mailings to teachers who requested remails when prompted 
for their survey. 

Also in mid-April, NORC sent letters to principals and district officials reminding them of the study and 
asking them to call the study’s toll-free number to complete an interview. Each type of respondent was 
assigned a prompting letter or a refusal letter based on disposition codes given to each case by the TIs. 

The mailing as a whole included approximately a quarter of the original principal and district official 
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interviewers had been trying to reaeh them and, again, to eall the toll-free number to set up an interview. 
Over one fifth of the original prineipal and distriet offieial samples reeeived this letter. 

Incentive Distribution 

Round III ineentive distribution proeedures sought to automate many proeesses that had been done 
manually during Round I of NLECSR. Just as in Round I, AIR eoordinated the aeeounting and 
distribution of ineentive eheeks to teaehers and sehool eoordinators. AIR again mailed all incentive 
checks to the respondent’s school address on file, unless the respondent specifically notified NORC or 
AIR with a valid reason for why they could not retrieve their check from the school. In addition, NORC 
ordered giant blue plastic paperclips with the NLECSR logo and contacting information to include as a 
small incentive with mailing materials. 

Case Review 

As in Round I, NORC staff addressed cases where teachers reported being OOS or received the “wrong” 
type of questionnaire due to inaccurate roster information or other similar reasons. Again, NORC 
recorded all information concerning OOS cases and presented it to AIR for analysis purposes. In this final 
round, one field manager was retained following the end of data collection to verify alleged OOS cases 
with the school where the information was unclear, as well as where a teacher’s status was unknown. 
Where respondents received and completed the wrong questionnaire, NORC sought to retrieve the 
subject-specific information. In early May, NORC worked with AIR to produce incentive checks for 
cases like these to include in a “retrieval” package which included photocopies of the relevant 
subject- specific pages to be completed, a BRE and their incentive check. Since the respondents had 
already completed a substantial amount of the questionnaire, the inclusion of the incentive check in the 
retrieval mailing was believed to encourage respondents to respond to the request. 

During data collection it was discovered by the TIs that some former district official respondents were no 
longer at the district and the position that they had held was vacant or rolled into another official’s 
responsibilities. At this point, where a position’s responsibilities were held by another (current) 
respondent or no one, the cases were determined in consultation with AIR to be OOS. In addition, it was 
determined by AIR that some current district official respondents were not suitable respondents for their 
purposes (i.e., their responsibilities did not include any areas related to the questionnaires). Thus, they 
were replaced by cases bearing the positions named by AIR. Changes like these made to the sample were 
detailed in the final delivery supplemental files. Due to various decreases and increases in the district 
official sample, the number of total respondents decreased from the original 65 to 60. 

Data Review and Delivery 

NORC delivered the Round III data in two parts: CATI data (from principal and district official cases) on 
August 10, 2004, and the PAPI data (from teacher cases) on August 19, 2004. In all, NORC delivered 45 
district cases, 482 principal cases, and 3,877 teacher cases (125 of which were OOS). There was a second 
delivery of teacher data on September 30, 2004. Per AIR, this delivery included 19 new cases that came 
in after the first data delivery was completed. Tables 2.3 and 2.4 report response rates for Round III. 

A number of district official positions were unmanned, eliminated, and inherited by other persons. In 
addition, AIR made some decisions on respondents in R3 which resulted in new cases. These changes are 
reflected here; 1 1 districts being affected. 
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Table 2.3. Round III response rates: Principals and district officials 



District 


Total R 
Released 


Total in 
Scope 
R Released 


% Overall 
Complete 


Total 

District 

Off. 

Complete 


% District 
Off. 

Complete 


Total 

Principals 

Complete 


% 

Principals 

Complete 


E-1* 


15 


15 


86.70 


3 


100.00 


10 


83.30 




59 


59 


79.70 


2 


66.70 


45 


80.40 


E-3 


38 


38 


71.10 


2 


66.70 


25 


71.40 


E-4 


26 


26 


73.10 


3 


100.00 


16 


69.60 


E-5* 


56 


55 


74.50 


1 


50.00 


40 


75.50 


E-6** 


42 


42 


64.30 


1 


33.30 


26 


66.70 


N-1* 


83 


83 


85.50 


2 


50.00 


69 


87.30 


N-2" 


21 


21 


81.00 


1 


33.30 


16 


88.90 


N-3 


44 


44 


70.50 


2 


66.70 


29 


72.50 


N-4 


20 


18 


77.80 


1 


50.00 


13 


76.50 


N-5*" 


30 


29 


72.40 


2 


100.00 


19 


70.40 


S-1 


4 


4 


100.00 


3 


100.00 


1 


100.00 


S-2" 


51 


50 


68.00 


2 


100.00 


32 


66.70 


S-3** 


68 


68 


76.50 


2 


66.70 


50 


76.90 


S-4* 


25 


25 


72.00 


3 


100.00 


15 


68.20 


S-5" 


30 


30 


66.70 


3 


100.00 


17 


63.00 


W-1 


8 


8 


87.50 


2 


66.70 


5 


100.00 


W-2 


19 


19 


89.50 


3 


100.00 


14 


87.50 


W-3 


25 


25 


80.00 


2 


66.70 


18 


81.80 


W-4* 


16 


16 


87.50 


3 


100.00 


11 


84.60 


\N-5” 


16 


16 


81.30 


2 


66.70 


11 


84.60 


Total 


696 


691 


76.30 


45 


75.00 


482 


76.40 



# Eleven total partial principal cases are included here. 

$ Two total partial district official cases are included here. 

* Eighteen schools closed for the 2003-04 school year and thus the principal cases for those have been made OOS, 
reducing the sample to 631 . 



Table 2.4. Round III response rates: Teachers 



District 


Total 

Quexes 

Mailed 


OOS 

Total 


Total 

Quexes 

Complete 


% 

Complete 

Overall 


B 


% ENG 
Complete 


Total 

MATH 

mailed 


% MATH 
Complete 


E-1 


90 


4 


64 


74.40% 


41 


72.50% 


49 


76.10% 


E-2 


433 


43 


236 


60.50% 


238 


60.80% 


195 


60.10% 


E-3 


401 


48 


252 


71 .40% 


194 


70.00% 


207 


72.70% 


E-4 


277 


36 


178 


73.90% 


136 


74.60% 


141 


73.10% 


E-5 


497 


38 


335 


73.00% 


226 


72.20% 


271 


73.60% 


E-6 


380 


41 


237 


69.90% 


186 


71 .50% 


194 


68.40% 


N-1 


802 


68 


510 


69.50% 


396 


71 .40% 


406 


67.60% 


N-2 


162 


10 


119 


78.30% 


86 


77.20% 


76 


79.50% 


N-3 


256 


17 


145 


60.70% 


136 


61.10% 


120 


60.20% 


N-4 


130 


17 


86 


76.10% 


65 


73.70% 


65 


78.60% 


N-5 


137 


11 


87 


69.00% 


66 


67.70% 


71 


70.30% 


S-1 


10 


0 


10 


100.00% 


5 


1 00.00% 


5 


1 00.00% 


S-2 


550 


34 


397 


76.90% 


279 


78.80% 


271 


75.00% 


S-3 


542 


36 


371 


73.30% 


279 


73.90% 


263 


72.70% 


S-4 


162 


8 


111 


72.10% 


88 


71.10% 


74 


73.20% 


S-5 


202 


14 


139 


73.90% 


103 


74.50% 


99 


73.40% 



m 
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District 


Total 

Quexes 

Mailed 


OOS 

Total 


Total 

Quexes 

Complete 


% 

Complete 

Overall 


Total 

ENG 

mailed 


% ENG 
Complete 


Total 

MATH 

mailed 


% MATH 
Complete 


W-1 


57 


18 


36 


92.30% 


27 


94.70% 


30 


90.00% 


W-2 


82 


3 


51 


64.60% 


40 


68.40% 


42 


61.00% 


W-3 


449 


18 


297 


68.90% 


235 


67.60% 


214 


70.40% 


W-4 


68 


1 


57 


85.10% 


37 


78.40% 


31 


93.30% 


W-5 


68 


1 


53 


79.10% 


36 


77.10% 


32 


81.30% 


Total 


5755 


466 


3771 


71.30% 


2899 


71.50% 


2856 


71.10% 



Note: OOS teachers have been taken out of the denominator. 

Case Study 

As a complement to survey work, NLECSR also included case study work to provide an in-depth 
description of the processes associated with CSR implementation. In particular, the qualitative NLECSR 
component worked to answer the following question: What supporting conditions and strategies are 
necessary to effectively implement and sustain reform in schools and at district levels? 

The NLECSR qualitative data collection activities occurred at the classroom, school, and district levels. 
These activities involved visits to 24 CSR model schools in five districts. Schools and districts were 
selected from the NLECSR subsample primarily based on school and district setting characteristics, such 
as achievement, duration of CSR model implementation, and student demographics. All schools and 
districts were visited during the 2002-03 school year. Selected CSR model schools in three districts were 
visited a second time during the 2003-04 school year for more in-depth data on CSR processes. 

On-site visits to schools and districts, semi-structured interviews and focus groups with school 
administrators and teachers constituted core data collection activities. Interviews were conducted with 57 
administrators (principals, assistant principals, and facilitators), 192 teachers, and 30 teacher focus 
groups. Individual interviews of administrators lasted approximately 60 minutes. Individual interviews of 
teachers and focus group interviews of teachers ranged from 15 to 30 minutes. All interviews were 
audiotaped and transcribed. To supplement these interviews, other data collection activities included 
school and classroom observations, collection of key documents such as school improvement plans, and 
interviews with community and district leaders. 

Analysis of the data collected involved two main activities. First, interview and focus group transcripts 
were coded by using a construct key mainly covering CSR model components (i.e., school culture, 
organization and governance, curriculum and instruction, assessment, and professional development). 
Throughout the coding process, researchers discussed the codes and defined them based on the data. Once 
all transcripts were coded, coded text was organized by using NUD*IST v.6, a qualitative software 
program. Using these coded data, case reports were written for each school and developed within- and 
cross-case data displays (Miles & Huberman, 1994). 

Second, we developed a rubric to facilitate the systematic analysis of school-level support for CSR and 
related constructs. The NLECSR Analytic Rubric contained four primary sections related to participant 
understanding of CSR, perceptions of the CSR model, school-level processes related to the model, and 
professional resources. To complete a rubric for each school, a researcher read all principal, facilitator, 
and teacher interview and focus group transcripts, identified text that informed the constructs related to 
understanding of CSR, and rated each respondent’s comments for both constructs. 
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Extant Achievement Data 

Extant school-level achievement data were collected for all schools in the study from the National 
School-Level State Assessment Database. Ideally, we would have collected extant student-level data for 
all schools as well. Not all states, however, administer state assessments in such a way, nor do they store 
student level data in such a way as to make this a worthwhile endeavor. We aimed instead to identify 
which of our 2 1 districts could provide 5 years of student-level achievement data that could be 
longitudinally linked, for achievement tests that were vertically equated. We identified Dodgeland, 
Riverton, Elm County, Hickoryville, and Rainfield as having appropriate data available, as well as a 
substantial number of NLECSR schools and comparison schools. 

Districts were asked to provide individual student-level data for students in all K-8 grades where test 
scores were available for 5 school years, from 1999-2000 through 2003-04. The five districts were able 
to provide data that allowed us to link a student’s records from one year to records from other years, 
which made it possible to do an analysis of achievement growth. Districts also supplied demographic data 
on their students and schools, where possible. Districts were asked to provide the data listed in Figure 2.1. 
Table 2.5 illustrates the cohort analyses these data make possible. 

Table 2.5. Cohort analyses possible with these data 



School Year 


Background 

1999-2000 


Background 

2000-2001 


Yean 

2001-02 


Year 2 
2002-03 


Year 3 
2003-04 












K 










K 


1 


Cohort K 






K 


1 


2 


Cohort 1 




K 


1 


2 


3* 


Cohort 2 


K 


1 


2 


3 


4 


Cohort 3 


1 


2 


3* 


4 


5* 


Cohort 4 


2 


3 


4 


5 


6* 


Cohort 5 


3 


4 


5* 


6 


7 


Cohort 6 


4 


5 


6* 


7 


8* 


Cohort 7 


5 


6 


7 


8 




Cohort 8 


6 


7 


8* 







* Indicates teacher survey data available in that year. 



Throughout this report we use pseudonyms for schools and districts in order to protect the confidentiality of 
respondents. 
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Figure 2.1. Student-level data requested 



1999-2000 and 2000-2001 

Individual student-level data for students in all K-8 grades where test scores are available: 

Student ID (linkable over time if possible) 

School ID (CCD ID if possible) 

School year 
School name 
Grade level 

Math test score (scaled score preferred) 

Reading test score (scaled score preferred) 

2001-02, 2002-03, and 2003-04 

Individual student-level data for students in all K-8 grades where test scores are available: 

Student ID (linkable over time if possible) 

School ID (CCD ID if possible) 

School year 
School name 
Grade level 
Date of birth 
Gender 

Race (Black, Hispanic, Asian, White, Other) 

English proficiency 
Special Education (lEP) status 

Free/reduced-price lunch status (or some other measure of socio-economic status) 

Gifted program (yes or no) 

Summer school attendance (yes or no) 

Days absent 

Math test score (scaled score preferred) 

Reading test score (scaled score preferred) 

School-level data for all students in K-8: 

School ID (CCD ID if possible) 

School name 
Enrollment 

Number of Students leaving the school during the year 

Number of Students entering the school during the year 

Number of Special education 

Number of Free/reduced price lunch status 

Number of LEP/ELL 




AMERICAN 
INSTITUTES 
FOR Research ‘ 





Chapter 3: The State and District Context for 
Comprehensive Schooi Reform 

Andrea Boyle 

Kerri Thomsen 

Kerstin Carlson LeFloch 



Introduction 

Although comprehensive school reform operates as a school-level process, it does not occur within a 
vacuum. Rather, CSR schools are embedded within districts, which in turn are embedded within states, 
both of which enact policies and engage in practices that may support or hinder CSR efforts. For CSR 
implementation to be strong and sustained, scholars have argued that it must take place within the context 
of a “supportive operating environment,” including the district and state (Bodilly, 1998). 

While most of this report is framed around the notion of a CSR “life cycle,” we first situate the 
school-level activities undertaken as part of CSR within their broader contexts. The primary objective of 
this chapter is to review the contextual background of district and state activities, both within the CSR 
research literature, as well as the specific states and districts that make up the NLECSR sample. In doing 
so, we will address Research Question 3 : What supporting conditions and strategies are necessary to 
effectively implement and sustain CSR models in schools and school districts? 

The organization of this chapter is as follows. First, we review the literature on the state and district roles 
in the implementation of CSR and offer a framework for understanding the key components of those 
roles. We then present our analyses of NLECSR’s state-level data collected from interviews with state 
CSR coordinators and a review of relevant policy documents. Next, we examine the district context for 
CSR, drawing on data from district interview respondents as well as NEECSR’s 2002-03 district survey. 
Finally, we explore some of the relationships between states’ and districts’ CSR activities. Throughout, 
we frame our discussion around three themes: (1) the alignment of CSR with other improvement 
strategies, (2) state and district activities to support CSR, and (3) the ways in which states and districts are 
refining their approach to CSR. 

Conceptual Framework and Literature Review 
States 

Although states have been increasingly active with regard to school improvement policy since the 
movement toward standards-based reform (Eusi, 1997), most state-level CSR policies typically did not 
originate until the Comprehensive School Reform Demonstration Program was created in 1998 as part of 
Public Eaw 105-78. This program (frequently referred to by its sponsors’ names, Obey-Porter) charged 
state education agencies (SEAs) with administering federal CSR grants to schools and districts. Federal 
policy and guidance for the new program allowed states considerable latitude in disbursing the funds. 
Although states needed to comply with certain minimum requirements, they could specify award 
priorities, application procedures, and funding allotments. 

The establishment of the Comprehensive School Reform Demonstration Program in 1998 extended the 
federal government’s support for schoolwide capacity-building reforms by offering grants of no less than 
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$50,000 for schools implementing CSR models. In 2001, Congress ineorporated the federal CSR program 
into the aeeountability-foeused NCLB. Furthermore, NCLB’s new polieies for the CSR program 
explieitly tied it to states’ aeeountability mechanisms; for instance, these new CSR policies included 
requirements that SEAs (1) give priority to schools identified for improvement or corrective action under 
their accountability systems (Title I, Part F, section 1604) and (2) annually evaluate the extent to which 
schools’ implementation of CSR has led to improved student performance (Title I, Part F, section 1607). 
Thus, CSR became framed as a capacity-building tool within an overall model of accountability and 
school improvement. 

As a result, states developed CSR strategies that were suited to their existing SEA and state policy 
contexts, and the resulting state activity became the subject of some limited research on the role of the 
state in CSR implementation (Flamann & Eane, 2004; Eane & Gracia, 2005; Datnow, 2005). From this 
research, three key themes emerge: 

1 . The evolution of the state role from a compliance enforcer to that of a capacity-builder. 

2. The alignment of CSR policies with the existing policy context 

3. The refinement of state policies to better support CSR 

While current research into states’ roles in supporting CSR is in rather short supply, this line of inquiry 
taps into more generalized theories that SEAs are moving away from traditionally limited roles as 
compliance enforcers to become more active facilitators of school and district capacity building through 
such activities as building technical assistance and professional development infrastructures, organizing 
resource allocation, and establishing reform frameworks, standards, and goals (Eusi, 1997; Eittle & 
Houston, 2003; Massell, 1998). 

The alignment of CSR and other state policies is another prominent theme in the current literature on state 
implementation of CSR. Since federal CSR policy affords states considerable flexibility in deciding how 
to implement the grant program, many SEAs made key policy decisions to adapt their CSR strategies and 
fit them into existing state accountability practices, initiatives, reform goals, and institutional contexts 
(Eane & Gracia, 2005; Hamann & Eane, 2004). In some cases, states have made direct choices to 
incorporate CSR into other state programs; Hamann and Eane (2004) describe how Maine merged the 
CSRD funding opportunity into a state-initiated high school reform program by adapting federal CSRD 
policy to target only high schools and by connecting the application for CSRD funding to the application 
for the state program funding. Moreover, states have employed components of their statewide systems of 
support for low-performing schools to facilitate and assist in schools’ CSR efforts (Eane & Gracia, 2005). 

Additionally, states’ decisions to incorporate CSR into broader school improvement strategies resonates 
with theories of systemic reform which advocate integrating state education policies to align them around 
a common set of educational objectives (O’Day & Smith, 1993; Fuhrman, 1993; Elmore, 1993; Eusi, 
1997). Such integration is intended to promote coherence across all levels of education systems and to 
replace “the practice of fashioning a separate program for each educational problem” whereby “individual 
projects, no matter how uniquely worthy, seldom reinforce one another and frequently send different, 
even conflicting messages to schools” (Fuhrman, 1993, p. 7). 

Districts 

In contrast to the state, the central role of the district has been the subject of more intensive scrutiny on 
the part of researchers. For example, in their study of Success for All (SFA) implementation. Cooper, 



AMHRICAN 
INSTITUTES 
FOR Research 






Chapter 3: The State and District Context for Comprehensive School Reform 



Slavin, and Madden (1998) argue that, while distriet partieipation is not necessary for initiating 
comprehensive school reform, it is “prerequisite to sustaining fundamental change in schools” (p. 399). 

Several important CSR studies have developed taxonomies of district strategies and conditions that will 
most optimally support CSR. In one of the more comprehensive studies of the district role in CSR, 
Bodilly’s study of the implementation of New American Schools (NAS) designs (1998) determined that 
certain district contexts and activities were associated with higher levels of implementation of CSR 
models. Districts that offered higher levels of support had higher levels of implementation while less 
supportive districts had lower levels of implementation. Using interviews in which school staff were 
asked which actions, policies, and conditions at the district level supported their reform work, Bodilly 
established the following framework for effective district-level support: 

1 . Observed leadership support and central placement of the initiative — Central placement of the 
initiative was measured in terms of the consistency of attention to the effort, the actions taken to 
ensure that school staff understood its importance, the lack of competing initiatives or setting a 
clear priority for this initiative, and the perceived longevity of the superintendent or main 
advocate 

2. Lack of crisis situations — Bodilly reports that school staff in districts facing crises tended to adopt 
a “wait and see attitude” and to assume “this too will pass” more often than staff from other 
districts. 

3. Culture of cooperation and trust — This component involves the history of a district’s relationship 
between management and labor. Examples of distrust reported by school staff included their 
perceived lack of commitment from district superiors, changes in policy, and perceived lack of 
follow-through. 

4. School level authority and/or autonomy — Schools without substantial decision-making authority 
often encountered problems where rigid district policies interfered with the implementation of 
their CSR model’s various components. Districts could help minimize such conflicts by affording 
schools significant control over curriculum, instruction, schedules and materials; personnel 
hiring, firing, transfer, and positions, professional development; and budget. 

5. Availability of resources for transformation — ^A lack of resources determined necessary by the 
design team was seen as a barrier to implementation, and some school staff interpreted the 
underfunding they observed as a message that the CSR effort was less important than other 
concerns. 

6. Design-compatible accountability and assessment systems — This component involves the 
compatibility between a CSR design’s requirements for student learning and the state and/or 
district requirements for measured student learning. 

Berends (2001) used this framework for district support in his quantitative analysis of NAS’s Scale-up 
period. As hypothesized, schools within districts that ranked a standard deviation higher on this support 
index had implementation levels that were about one-tenth of a standard deviation higher than those in 
less supportive districts. This study also found that, on average, schools reporting more resources for 
implementation (including materials; professional development; time for planning, collaboration, and 
development; consultants to provide ongoing support; technology; and funding) had higher levels of 
implementation. 

A policy brief by the Consortium for Policy Research in Education (1998) argues that districts play 
several crucial roles in initiating and sustaining CSR programs. While the RAND work (Bodilly & 
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Berends, 1 999) described supportive district conditions, the CPRE literature described specific district 
activities that would support CSR. These included (1) assisting schools in selecting a model design, (2) 
structuring the CSR approach into the school’s and district’s continuing operations, (3) designing a new 
district operating environment, (4) finding a district-appropriate method of supporting CSR, (5) 
monitoring and controlling the quality and performance of design teams, and (6) creating an outreach 
process that engages and informs parents and the school’s community. 

Slavin (1998; cited in Maciver & Balfanz, 2000) suggested districts perform the following functions in 
facilitating comprehensive school reform: (1) identify the set of models that will be offered to schools; 

(2) determine what the district will do to support implementation; (3) determine how the models fit with 
the district’s overall reform strategy; (4) identify schools that will have opportunity to implement models; 
(5) assist schools in learning about each model so that schools can make an informed choice; (6) ensure 
that each school works closely with the model developers; and (7) design a process for evaluating a 
model’s success. 

Datnow and Stringfield (2000) indicate that districts need to establish a finite set of widely shared goals 
tied to long-term key improvement measures; create a coordinated and broad-based plan for 
disseminating information about reform; facilitate a thoughtful, critical evaluation of what needs to 
change in a school and why; align policy systems to support reform; and be willing to change to adapt to 
the needs of the reform. 

Although the district literature is more diverse than are studies of the state role, we can discern at least 
two broad themes. First, districts may vary with regard to the extent to which CSR strategies are aligned 
with existing district strategies to support school improvement, including curricular strategies, 
accountability policies, and professional development. Next, districts vary with regard to the level of 
support they provide to CSR schools, including support during the model selection process, support 
during implementation (including contact with model developers), appropriate leadership, adequate 
resources, and providing school-level autonomy. 

To best integrate NLECSR data on the state and district roles in CSR, we sought to distill our discussion 
into three primary themes, which reflect both existing research and our own findings. The first of these is 
policy alignment: both the state and district literature clearly underscore the importance of ensuring an 
appropriate alignment or coherence between existing policies (including, for example, curriculum and 
professional development) and CSR initiatives. The second theme is that of support strategies : at the state 
and district levels alike, the literature points to specific policies and strategies to support the selection and 
implementation of CSR models. Finally — although this is less prominent in the current literature — 
researchers suggest the importance of a feedback loop (Orland & Goettel, 1982; Datnow & Stringfield, 
2000; Berends et al., 2002); that is, a mechanism through which education administrators improve upon 
existing policies, perhaps through a formal evaluation process or by soliciting and responding to feedback 
from stakeholders. In this chapter, we will refer to this theme as that of policy refinement (Figure 3.1). 

Beyond the overarching question — what supporting conditions and strategies are necessary to effectively 
implement and sustain CSR models in schools and school districts? — this chapter also addresses several 
specific subquestions associated with the state and district roles in supporting CSR. First, with regard to 
the state level, we consider the extent to which CSR is integrated with other state strategies for school 
improvement, the types of support that are provided by states, and the ways in which states are refining 
their approach to CSR. Then, we turn our attention to the district and ask how does CSR connect to the 
district’s overall improvement strategies? In what types of CSR support strategies do districts most 
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frequently engage? How have districts’ approaches to CSR evolved? Finally, we inquire how states’ CSR 
strategies interact with CSR policies of the districts nested within them. 

Figure 3.1. Conceptual framework 



State or District Improvement Strategies 



I 

Feedback/ 

Policy 

Refinement 






CSR Strategies: 

• Support for 
model selection 

• Support for 
implementation 




Alignment 






Improvement 

Policies: 

• Accountability 

• Curriculum 

• Professional 
Development 



CSR Model Implementation 



State Methods/Analytic Approach 
Data 

To investigate state strategies toward comprehensive school reform, we focused on two primary data 
collection strategies in the 16 states in which the NLECSR districts and schools were embedded. The first 
involved a thorough review of relevant state and federal policy documents. These included the policy and 
guidance for Title I, Part F of NCLB, which reauthorized the federal CSR grant program initially 
established in 1997. We also examined states’ Consolidated State Applications under NCLB, in which 
states were required to detail their overall approach to the federal CSR program, targeting strategy, and 
evaluation methods. In addition, we consulted policy documents available from state department of 
education Web sites. Among these were CSR grant RFPs, evaluation rubrics for grant applications, CSR 
component descriptions and crosswalks, and agendas for technical assistance workshops. 

Although these documents provided insight into the state role with regard to comprehensive school 
reform, they lacked the depth necessary to address some of our more fundamental questions. We 
conducted interviews with state officials with primary responsibility for the administration of the federal 
CSR program. We developed an open-ended interview protocol with questions that probed the following: 

♦ History of state involvement in CSR 

♦ Evolution of state strategy with regard to CSR 

♦ Role of CSR in state system of support for low-performing schools 
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♦ Rationale for funding strategy 

♦ State technical assistance 

♦ State role in model adoption 

♦ Challenges associated with CSR 

Although we had anticipated that the interview would last approximately 30 minutes, respondents often 
spoke longer, and several interviews lasted 45 minutes. One staff member conducted the interview, while 
a second listened and took notes on computer. These notes were later reviewed in conjunction with the 
audio recording and were supplemented as necessary. From each interview we produced a transcript that 
was close to verbatim. 

Analytic Models and Methods 

Staff members who had developed the interview protocol were also those who had conducted the 
interviews, increasing familiarity with the interview content and facilitating analyses. First, we reviewed 
each transcript, highlighting key passages related to states’ CSR policy-making decisions. Upon a second 
review, we began to associate codes with highlighted text and entered passages into an excel template 
which facilitated the comparison of common themes across states. 

Through a process of identifying codes, clustering conceptually linked variables, refining categories, and 
creating indices, we developed an analytic mechanism for determining patterns in state CSR activities. In 
short, we identified three dimensions to describe states’ CSR policy activities, reflected in the conceptual 
framework presented earlier: 

♦ The alignment of CSR dimension consisted of state policies and activities to (1) target certain 
types of schools for CSR funding and (2) connect CSR to the state’s system of support for 
low-performing schools. Connections between CSR and a state’s system of support might occur 
through collaboration among CSR coordinators and state school support staff or through states’ 
incorporation of CSR into policies governing their school improvement strategy. 

♦ The support for CSR dimension included four components: (1) using the CSR planning 
application/planning process to ensure schools’ readiness to implement a CSR plan, (2) allotting 
funds to schools, (3) providing technical assistance to help schools write successful grant 
applications, and (4) providing technical assistance to help schools implement their proposed 
CSR plan. 

♦ The refinement of CSR dimension involved only one component: the degree to which states 
change their CSR policies to meet state and school needs. 

For each of the seven components, we developed a set of categories and descriptors to capture the 
variation among states’ approaches to them. We then assigned each approach category a numeric rating 
such that a score of 0 indicated an approach heavily dictated by federal policy and/or guidance, and higher 
scores indicated approaches that involved higher levels of activity at the state level. 
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State Findings/Results 

Alignment of CSR With Other State Strategies 

State System of Support: Connections with CSR 

Under NCLB, all states are required to establish a system of support for sehools identified for 
improvement under Title I. Frequently, these systems extend serviees beyond only Title I sehools 
identified for improvement and eneompass non-Title I schools, schools determined to be low-performing 
under the state accountability system, and schools that do not make adequately yearly progress for only 1 
year. NCLB provisions articulate specific structures that states should establish to support identified 
schools, including school support teams, distinguished teachers, and distinguished principals. Despite 
relative uniformity of federal guidelines, states vary greatly in the ways in which they have 
conceptualized their systems of support. 

Because we have assured confidentiality to all participants in our study — from the state to the local 
level — we cannot describe state systems in a way that would reveal any states. Flowever, because there 
are commonalities among the states in our study, and indeed throughout the country, we can outline 
general trends and “types” of states. The examples presented below are hypothetical composites of the 
strategies we found among the states within our sample. 

Among the states in our study are several that have spent many years developing comprehensive support 
systems. Often, these systems include both tools and structures that together encompass an overarching 
improvement process. These tools are designed to assist schools in the needs assessment process, data 
analysis, developing a school improvement plan, aligning funding streams, and monitoring 
implementation. The structures may consist of school support teams, school improvement coaches, or 
regional agencies that provide assistance to schools on an as-needed basis. The core of such systems is a 
coherent vision for school improvement, generally grounded in research on school reform. States that 
have such systems weave new federal grant opportunities into the overall approach to school 
improvement — and such was the case with the federal CSR program in nearly half of the states in our 
study. 

Let us consider an example of a state with a coherent and comprehensive system of support for identified 
schools. Often, states with such systems conduct a triage of schools based on core accountability 
measures, determining which states had missed the highest number of Adequate Y early Progress (AYP) 
targets or which had failed to make AYP for several consecutive years. The state then schedules for each 
of these schools to be visited by a school support team. The team may spend a week at a school, helping 
administrators and teachers to review assessment data, identifying content areas that presented challenges 
or subgroups of students that could benefit from targeted interventions. The school support team is most 
often comprised of SEA school improvement staff, most of whom also have responsibilities for other 
federal programs, including Title I Part A, CSR, and perhaps Reading First or supplemental educational 
services. Such staff members assist the school as they identify appropriate improvement strategies and 
funding opportunities. Working with school staff, the SEA support team help identify which of these 
high-need schools had adequate capacity to benefit from a federal CSR grant, whether the CSR program 
would integrate well with any existing school-level initiatives, and whether the staff had the capacity to 
write a CSR grant. As an official in one such state explained. 



For an overview of No Child Left Behind’s provisions regarding statewide systems of support, see No Child Left 
Behind Act of 2001 (2002). Pub. L. No.107-110, §1117, 115 Stat. 1498-1501 (2001). 
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CSR falls into and underneath the sehool improvement division, so we are not only in the 
same office with all of the low-performing schools, we are also part of the process that 
goes out and facilitates the assistance to those schools. . . we make sure that school 
improvement plans are aligned with all of the other initiatives in the school as well as the 
technical assistance and guidance we provide. 

In this context, CSR is perceived as an important support strategy (among others) to be targeted to 
appropriate schools as part of a broad state vision for school improvement. 

At the other extreme are states that have devoted fewer resources to the development of a system of 
support. There are many reasons for which states have failed to establish such systems, including 
insufficient human or financial resources, strong traditions of local control, or relatively few schools 
identified for improvement. In this context, CSR is merely a funding stream to be administered by the 
state, rather than an opportunity to be leveraged as part of a broader strategy to address the needs of 
low-performing schools. In the sample of states we interviewed for this study, this constituted 
approximately one third of the states. 

Targeting CSR Funds 

States also vary in important ways with regard to the way in which they target schools for CSR grants, 
and this variation is often — ^but not always — associated with the coherence of the overall system of 
support. Federal CSR guidance articulates CSR eligibility requirements, but these are relatively broad. 
Formally, local education agencies (LEAs) apply to states on behalf of Title I-eligible schools, preferably 
those in which there is the greatest likelihood of promoting schoolwide change and improved student 
achievement. Among the states in our sample, approximately one third adopted these fairly general 
targeting strategies: eligible schools were high-poverty, low-performing schools that had missed AYP or 
been identified for improvement or corrective action. These states did little to focus the pool of potential 
CSR schools. 

However, federal guidance also permits states to set their own more explicit eligibility requirements, and 
many states have further refined their provisions to focus on a specific subset of schools. Such was the 
case in half of the states in our sample. In states with highly focused eligibility rules, state officials 
described the evolution of their approach as part of an overall policy refinement, intended to encourage 
substantive and sustainable reform. Some eligibility restrictions were revised to encourage school-level 
programmatic coherence: for example, some states limit Reading First schools’ participation in CSR to 
avoid overwhelming them with program requirements. One state official described. 

When you have a low-performing school. . . and we overload them with any number of 
grants, it may be CSR and Reading First or CSR and 2F‘ Century Schools, there is a 
limited amount of time in a workweek, and so when Reading First requires or mandates a 
certain amount of training or requirements, and when CSR has other requirements, one of 
them just falls by the wayside, and typically based on my experience that has been 
CSR. . . So I’m trying to coordinate now, and we’re going to prioritize those schools that 
don’t have existing reform s in their schools. 

Several states elected to focus on schools in “dire need:” those identified for restructuring or at the lowest 
level of state accountability designations. In contrast, other states focus on schools at the other end of the 
improvement spectrum, targeting schools that were only recently placed on academic watch or probation. 
As one state official explained, “There’s a far greater benefit for schools that have had a readiness to 
benefit as opposed to taking the lowest performing schools, which was what we’ve done in the past.” 
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These schools had not yet been identified for improvement, and CSR was viewed as a mechanism to stave 
off potential identification in subsequent years. 

In a few interesting cases, states opted to target schools that were Title I-eligible (but not receiving Title I 
funds) and identified for improvement. These schools, while still impoverished, did not benefit from Title 
I dollars so CSR was perceived as an effective mechanism to ensure these schools had access to another 
school improvement strategy and the associated supports. One state official recalled, “This year we said, 
‘we’re giving a lot of money to Title I schools, so who are we going to address with CSR money to?’ So 
this year we decided to give the money to schools which are Title I eligible, but are not Title I served, and 
in need of improvement under NCLB.” Finally, a few states focused eligibility on specific levels — only 
high schools, or only middle schools — or a specific subject. 

Support for CSR 

Using the Application/Planning Process to Build Capacity 

In addition to formal eligibility requirements, many states were concerned with targeting schools that had 
at least a minimal level of capacity to support reform, or possessed a “readiness to benefit” from the funds 
they would receive. One of the most common and fundamental challenges states encountered in 
developing their approaches to CSR was the fact that they were dealing with high-poverty, 
low-performing schools which often lacked the capacity to develop a comprehensive school reform plan 
that was (1) compatible with their particular needs and (2) capable of being successfully implemented. 
Federal policy restricted use of CSR grants to implementation support only and thereby limited schools’ 
access to resources they could use for needs assessment and planning. In many cases, SEAs found that 
schools were applying for CSR funds without possessing a clear, well-conceived understanding of how 
they would actually implement a CSR program and how that CSR program fit into other efforts taking 
place in their school. As one CSR coordinator explained. 

They’re low-functioning schools, and if they had staff with the skills and capacity to 
write grants and go through that process on their own, they wouldn’t be in CSR — they 
wouldn’t be low-performing. So, what happens is, they have an opportunity to apply for 
money, they have a limited time in which to apply (4, maybe 5 months), and they slop 
together a proposal. . .There is not a readiness on the part of the school; there’s isn’t a 
comprehensive look at what they are doing in a comprehensive way so that they can be 
good consumers of these funds and technical assistance. 

Many SEAs recognized early on that for CSR to serve as an effective capacity-building strategy in their 
state, the schools they were targeting typically needed assistance in creating an appropriate CSR plan and 
in preparing school-level stakeholders to implement that plan in an effective manner. To address this 
issue, some states established policies to promote collective stakeholder support for CSR; such policies 
included required levels of teacher buy-in and requirements for district representatives to attend state CSR 
meetings. 

For example, some states used statewide school improvement planning processes to aid in schools’ 
development of CSR plans. One state coordinator described CSR as part of the state’s system for creating 
Title I schoolwide improvement plans: “All the funding sources and all the efforts are integrated into one 
plan. CSR is part of that.” Because the state connected CSR to this process, applicant CSR schools benefit 
from state training in identifying the root causes of their achievement gaps and corresponding needs to be 
addressed in their CSR strategy. Other states were less successful in their attempts to incorporate CSR 
into their system for school improvement planning. One state targeted CSR funding toward schools 
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entering its speeial assistanee program, whieh provided eoaehing as well as needs assessment and 
planning assistanee to schools identified for improvement. When the requirements for the state program 
clashed with the requirements for CSR, schools became overwhelmed with excessive mandates and time 
constraints. At the time of our data collection, the state was in the process of adjusting this strategy such 
that schools would go through the state planning assistance program first to create a comprehensive 
improvement plan and then have the opportunity to apply for CSR funding to continue those efforts. 

Technical Assistance for Grant Writing 

In addition to adapting their application procedures, states created unique approaches to provide schools 
with technical assistance during their actual grant writing. Because the targeted schools typically begin 
the CSR process with very low capacity, they often lack access to skilled grant writers to complete the 
lengthy CSR applications. One CSR coordinator perceived the daunting requirements of these 
applications to be the principal reasons for which schools were not applying for CSR. “I’d start talking 
about CSR and what the requirements for that were, [and] people’s eyes would glaze over; it was 
daunting enough for them to write an improvement plan, and CSR just kind of put them over the edge,” 
she explained. 

To improve schools’ chances of writing successful grant applications, SEAs have developed a range of 
techniques to both familiarize schools with CSR as a general school improvement technique and to aid 
schools in writing thoughtful, high-quality CSR proposals. While nearly every state CSR program 
answered schools’ phone and email requests for information about the grant opportunity, about two thirds 
of the states in this sample also held statewide workshops to explain their CSR application requirements. 
One state with a particularly in-depth approach to grant-writing technical assistance offered a series of 
workshops to discuss the application requirements but also how to conduct a needs assessment, how to 
connect that needs assessment to a school improvement plan, and how to align state standards and 
policies to that school improvement plan. Several states also encouraged schools to bring drafts of their 
applications and facilitated peer review sessions to improve the applications’ quality. 

CSR coordinators from about one third of the states in our sample described working individually with 
schools to facilitate their grant-writing process. One CSR coordinator explains the use of individualized 
assistance as a means of guiding schools in the right direction: 

We find that a lot of schools really start out on the wrong foot. They start out telling you 
all the stuff that you really don’t need to know, and it doesn’t support anything. The more 
that you can work with them and say, “OK, this is what you are writing, but you need to 
go back to your staff and you need to do a lot more work”. . . If you can work with people 
along the way and just get them focused on the right strategies and methods they need to 
use to even put an RFP together — the kind of work they need to get stuff in writing 
down — I find that that’s very helpful when they start actually putting pen to paper. 

Many SEAs, particularly those conducting CSR competitions with large numbers of applicants, simply 
did not have the manpower to provide this type of individualized assistance, but some such states were 
encouraging schools to form partnerships with local universities and/or educational resource centers to 
provide the grant-writing assistance they cannot. 

Technical Assistance for Implementation 

State patterns in the provision of technical assistance for CSR often mirror their overall system of support. 
Among the states in our sample, several had systems of support that relied primarily on decentralized, 
regional support agencies. States that traditionally centralized most support strategies within the SEA 
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tended to do so for CSR aetivities as well. A few states eollaborated with regional edueational 
laboratories to provide technieal assistanee to CSR schools, but this was a somewhat less common 
approach. 




Although many states provided technical assistance for the grant writing, fewer states provided technical 
assistance to support the implementation of CSR. The most active form of technical assistance consisted 
of school-level site visits with the explicit purpose of determining progress in model implementation. 
Approximately one third of the states in our sample provided direct, one-on-one assistance to schools that 
were implementing CSR models through the federal program. A bit more co mm on (approximately one 
half of the states in our sample) were states in which SEA staff visited CSR schools in conjunction with 
another required site visit. Said one, “Every year, I visit each of these schools and I often try to coordinate 
with other activities since Em a Title I consultant.” 

In addition, the majority of states in our sample conducted workshops to support implementation of CSR 
strategies. These workshops focused on such topics as principal leadership, sustainability, 
coaching/reform facilitation, and assessment practices. While a less focused form of technical support, 
states often worked to ensure these workshops were meaningful for participants. For example, one state 
official noted, “We always have teams come, which is a good thing, we don’t ask just one person to 
participate, we usually have a team that can bring information back to the school. We really encourage 
whole school participation that way.” 

Interestingly, several states engaged directly with model developers when schools encountered problems. 
Several officials described cases in which model developers were not meeting the terms of the contract, or 
were not providing the type of support the school needed: “We were kind of the mediators, we worked 
between schools and model providers in case there were any problems.” In another situation, a state 
official determined that a school should have never opted to implement a specific model, but the model 
developer had offered to write the grant and the school agreed to go along. However, the school had none 
of the existing infrastructure necessary to implement the model, putting them in an untenable situation. In 
this case, the state requested that the model developer sever the contract and they complied. 

Most often, states had several layers of technical assistance, each of which provided a slightly different 
type of support. One CSR director described the set of technical assistance supports that his state 
provides: 

Eet’s take what happens after the moment a school gets an award: the first thing we do, 
prior to the start of their first school year, we do a one-day orientation with them into the 
comprehensive school reform process. And then, a little later in the summer, for the past 
several years we have done a two-day data retreat to get them familiar with the potential 
for data serving them in the decision-making process. . . . The goal annually from my part 
is to then visit at least two-thirds of the schools to make personal monitoring visits to 
them and to spend an hour and a half or so to discuss with their leadership team members 
talking about their implementation and concerns and issues. ... In our office we also 
have regional consultants. ... I have asked them in the past to select a couple of schools 
to shadow and make an occasional visit to get a sense of the quality of implementation 
and to find out what technical assistance they may need that we can provide. 

Finally, some states simply did not have the resources to provide technical assistance for implementation, 
either because of budget and staffing cuts, or a large number of CSR schools in the state — too large for 
them to provide adequate assistance. As one interviewee admitted, “We basically do not do any kind of 
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technical assistance. . . . The only thing that we do is basically when we’re on the phone, and they call and 
ask us questions.” 

Funding Strategies 

States had considerable decision-making authority in the manner in which federal grant money was 
allocated to schools and districts. Federal policy required each state-awarded subgrant be at least $50,000 
in size and restricts the amount of funding that could be reserved for states’ and districts’ administrative 
costs, but all other policies determining schools’ grant size is left to states. Of the states in our sample, 
only two chose to award schools the federal $50,000 minimum in their 2004-05 funding cohorts. Four 
states allowed schools to request their grant amount as long as it did not exceed a specified maximum. Six 
states based their funding decision on school characteristics such school size and/or determined need. 
Three states chose to reduce the size of their schools’ grants over the course of the 3-year funding cycle in 
an effort to gradually reduce schools’ dependence on federal funding. Such efforts to wean schools from 
the federal funding stem from the intent noted in federal policy that CSR grants are merely start-up funds; 
by the end of schools’ 3rd year of funding, their CSR implementation should be sustainable through the 
use of other local funds. 

State policies for CSR funding occasionally emphasized sustainability. Several states encouraged schools’ 
access to other funding sources during the application process. For instance, one CSR coordinator 
described using a budget that shows “what they’re going to be putting into these initiatives out of all the 
other funding sources in the school, and then we have totals at the bottom so we can show that CSR is just 
a small piece of all of the money that is going into that particular school.” In some cases, SEA staff 
performed extensive budget reviews to enforce policies that restrict schools’ use of CSR funding. For 
example, several states limited schools’ ability to use CSR funding to cover large budget items such as 
literacy or math coach salaries so that those resources are not lost once federal funding ends. Some states 
emphasized the issue of sustainability at annual implementation reviews. One coordinator, perceiving that 
some schools “don’t look past tomorrow if you don’t make them,” personally offered technical assistance 
in resource allocation and required schools to report on how they would incorporate other funds after the 
3 years of federal funding. 

Refinement of CSR Policy 

Although many states made efforts to adapt CSR to their state contexts, at the time of our data collection 
this remained an evolving process. Some states were on a policy trajectory of continuous refinement, but 
others seemed to be more stagnant, or in decline. Among the states we interviewed, we perceived three 
major themes with regard to their ongoing adaptation of state CSR policy. 

First, some states clearly were responding to lessons learned, and were continually refining their approach 
to CSR. For example, one state’s CSR staff came to believe that the model fairs they were conducting 
were encouraging schools to simply “try to buy something to fix their problems.” The CSR staff then 
adjusted its model fairs to focus more on the effective strategies that CSR models incorporate and less on 
the particular models themselves. The state also placed a heavier emphasis on needs assessment and 
data-based decision-making during its application process so that schools were “not just picking 
something that looks good.” 

Other states underwent a minor policy revolution, dramatically shifting the direction of their CSR policy. 
In most cases, such revolutions occurred when new SEA staff assumed the role of CSR coordinator and 
implemented significant changes to fit his/her vision for the program. Such changes in CSR leadership 
occurred in several states in our sample, and in most of these cases, the incoming CSR coordinator had 
previous experience working in the state’s school improvement or accountability division. Two such CSR 
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coordinators made efforts to better align CSR with the state’s other school improvement policies after 
perceiving the state’s earlier, less integrated approach to be ineffective. One of these coordinators 
described her work as “moving this huge ocean-liner here. ... I took over schools and just couldn’t figure 
out why they weren’t going anywhere with this;” she then orchestrated a series of changes in the state’s 
targeting strategy, application process, and technical assistance strategies to create a clearer connection 
between CSR and the states other school improvement programs. 

Finally, some states suffered a reduction in resources such that they could do very little to adapt CSR to 
their state context. In these cases, the state official simply hoped to comply with the requirements of the 
grant so that he/she could continue to provide resources to the school level, but held little hope that the 
SEA could do much in terms of supporting these schools. One such official described CSR as a 
“marginalized program” and explained “as long as I keep the reports going, nobody really seems to care 
what happens with it. In our state, it’s viewed as just another source of money for our schools.” 

District Methods/Analytic Approach 
Data 

The district analysis drew from two parts of NLECSR. First, it used qualitative data collected from 
interviews with district administrators in the following five districts (identified by pseudonyms): 
Oceanway, Riverton, Everville, Eastwicker, and Dodgeland. A small team of researchers conducted a 
total of 23 interviews in person at the district central offices over the course of a 2-year period. The first 
wave of interviews occurred between October 2002 and February 2003, and the second wave ran from 
February, 2004 through May, 2004. Interview respondents held a range of district-level positions and 
included chief academic officers, curriculum directors, CSR coordinators, and other personnel in such 
district offices as curriculum and instruction, professional development, data, and accountability. 
Researchers structured the interviews loosely around a protocol of open-ended questions that featured the 
following topics: 

♦ The district’s overall goals and strategies for school improvement 

♦ The district’s involvement in schools’ CSR adoption and model selection processes 

♦ District activities to support CSR 

♦ Changes that occurred at the district and school level in response to CSR 

♦ State and district policy alignment regarding CSR 

♦ Professional development, data-based decision-making, and resource allocation and their impact 
on CSR 

All interviews were tape-recorded, and verbatim transcripts were created from these audio recordings. 

The district analysis also incorporated data from the second wave of NEECSR’s district surveys which 
were administered to central office staff in all 21 districts participating in the NEECSR study. We chose 
to draw from the second round of survey data collection because it offered the most complete set of data 
and, occurring during the 2002-03 school year, it coincided with the initial round of district-level 
interviews. While the survey data featured in this analysis are largely quantitative in nature, verbatim 
responses to short, open-ended survey questions were also used. 
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Analytic Methods 

We began our analysis of the qualitative data by eare fully reviewing and coding district interview 
transcripts for themes relevant to CSR. We refined these themes through a series of subsequent reviews 
and isolated data related to the following key dimensions: (1) alignment of CSR with other strategies, 

(2) support for CSR, and (3) CSR policy refinement. We defined these three dimensions at the district 
level as follows: 

♦ The alignment of CSR dimension involved how CSR related to the district’s overall plan for 
instruction and school improvement. Specifically, it included how administrators perceived CSR 
to interact with other instructional policies such as content standards and curriculum requirements 
as well as the measures districts took to promote coherence between CSR and these policies. This 
dimension also encompassed policies that influenced schools’ decisions to use CSR over other 
improvement strategies and policies that affected schools’ selection of particular CSR models. 

♦ The support for CSR dimension included information regarding the activities districts 
performed to facilitate schools’ selection and implementation of CSR models. 

♦ The refinement of CSR dimension related to the evolution of districts’ policies and attitudes 
regarding CSR as well as the outlook for CSR’s future in the district. Topics that fell under this 
heading included how administrators adjusted their CSR policies in response to lessons learned, 
what they perceived to be the current focus of their CSR activities, and how they predicted their 
attitudes toward CSR would change in the future. 

After coding the data according to these dimensions, we examined the variation in the case study districts’ 
approaches to CSR and searched for patterns among their behavior. We also noted differences across 
districts to explore how administrators were shaping CSR policies in response to their particular district 
context. 

To identify patterns in CSR support strategies and model adoption policies across a wider sample of 
districts, we supplemented the qualitative investigation of our five case study districts with analyses using 
survey data from all 21 districts. First, we developed measures that rated districts according to (1) the 
extent to which they exerted control over schools’ CSR model adoption and selection through the use of 
mandates and (2) the range of different CSR support activities they offered to schools. To measure 
districts’ policies toward CSR model adoption, we created the following two scale variables: (1) a CSR 
Mandate variable to indicate the degree of district involvement in schools’ adoption of CSR models 
(i.e., whether the district reported no role in CSR, supported CSR, required some schools to do CSR, or 
required most or all schools to implement CSR) and (2) a CSR Model Selection Requirements variable to 
capture the extent to which district mandates restrict schools’ CSR model options (i.e., whether the 
district did not mandate CSR, mandated CSR but permitted the use of any CSR model, mandated CSR 
and required schools to select from a limited set of models, or mandated the use of a particular CSR 
model). After creating these district mandate measures, we read through responses for survey items 
probing why districts required schools to use CSR, including whether these mandates were issued when 
schools were identified for improvement or when schools failed to make AYP. 

To rate districts’ range of support activities, we created a Total Support variable which represented the 
proportion of 29 CSR support activities each district reported providing to schools. We also generated a 
set of seven additional support variables — Model Selection Support, Funding Support, Community 
Outreach Support, Networking Support, CSR Implementation Technical Assistance, CSR Evaluation 
Support, and Contact with Model Developers — which rated districts along categorical subsets of the Total 
Support measure (see Table 3.1 for a full description of each of these measures). 
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Table 3.1. CSR support activity rubric 



CSR Model Selection 
Support 
(6 items) 


District assists in needs assessment. 

District organizes model fairs. 

District provides travel funds to visit CSR schools “in action.” 
District provides written documentation on CSR. 

District provides one-on-one help in model selection. 

District helps schools identify models. 


Funding Support 
(6 items) 


District has funds for CSR schools. 

District redesigned budget system for school autonomy. 

District encouraged schoels to redeploy money frem categorical programs. 

District realigned spending to support CSR. 

District assists schools in securing resources for CSR. 

District provides financial incentives for professional development to support CSR. 


Community Outreach 
Support 
(4 items) 


District discussed CSR in an open forum. 
District provides communication to community. 
District efforts to involve community. 

District provides reform evaluation information. 


Networking Support 
(2 items) 


District facilitates meetings of principals with the same CSR model. 
District facilitates meetings of faculties with the same CSR model. 


CSR Implementation 
Technical Assistance 
(5 items) 


District provides instructional leadership. 

District facilitates model negotiations. 

District requires school improvement plan fecus on CSR. 

District increased professional development days to support CSR. 
District increased common planning time to support CSR. 


CSR Evaluation Support 
(4 items) 


District staff conduct regular site visits to monitor implementation. 

District receives written reports from schools implementing CSR. 

District has established timelines for key implementation activities. 

District disseminates data on how different models are working in the district. 


Contact with Model 
Developers 
(2 items) 


District staff meet with model developers at least once per semester. 

District staff communicate with model developers at least once per semester. 



The first, straightforward analysis involved creating a series of scatter-plot graphs to observe whether 
relationships existed between the CSR Mandate variables (independent variables) and each of the eight 
support variables (dependent variables). For example, we charted the value of districts’ CSR Mandate 
level in relation to the Funding Support scale, followed by all other support scales, resulting in eight 
different relationships. In most cases there was no apparent relationship. However, to each we added a 
simple regression line to calculate the extent to which a linear relationship existed between each set of 
variables; the resulting r^ value informed us of the strength of the relationship between these variables. 

Recognizing that nonlinear relationships might also exist among these measures, we supplemented our 
analyses by searching for patterns among districts with similar mandate and support policies. First, we 
clustered districts into groups based on their reported CSR mandate and CSR model requirement policies. 
We then compared the support activities reported among districts within each of these clusters to 
determine whether specific approaches to CSR mandates and model requirements had implications for 
district support activities. Finally, we repeated this technique in reverse by clustering districts according 
to high, medium, and low levels of support to check whether districts offering particularly many or 
particularly few types of support yielded any patterns with regard to mandate policies. 
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Lastly, we read through distriets’ responses to open-ended survey questions regarding their support 
practiees and reexamined interview data related to CSR support in seareh of additional detail coneeming 
districts’ support strategies. 

Measures 

CSR mandate 

The CSR mandate variable was calculated as a scale of 0-3, where a value of 0 indicated that the district 
played no role in CSR, a value of 1 indicated the district supported CSR but did not require it, a value of 2 
indicated the district required CSR in some schools, and a value of 3 indicated the district required CSR 
in most or all schools. 

CSR model selection requirements 

The CSR Model Selection Requirements was also created as a scale from 0-3, where a value of 0 
indicated that the district did not mandate schools to adopt a CSR model, a value of 1 indicated that the 
district mandated at least some schools to adopt a CSR model but placed no restrictions on model 
selection, a value of 2 indicated that the district mandated at least some schools to adopt a CSR and 
required them to select from a limited set of models, and a value of 3 indicated that the district mandated 
a particular CSR model in at least some schools. 

CSR support 

To measure the amount and types of CSR support each district provided, we created a rubric that included 
a set of 29 support activities organized into the following seven categories: Model Selection Support, 
Funding Support, Community Outreach Support, Networking Support, CSR Implementation Technical 
Assistance, CSR Evaluation Support, and Contact with Model Developers. For each category, districts 
received a score equal to the number of support items they provided divided by the total number of items 
in that category.'’ Thus, a district that indicated it provided 3 of the 6 support activities included under the 
CSR Model Selection Support category would receive a rating of 0.5. Each district also received a total 
support score which we calculated by dividing the total number of support activities a district reported by 
the 29 possible activities that we probed. It is important to note here that these measures indicate the 
range of CSR support activities that districts reported and do not capture the frequency or intensity of 
support provided. Finally, to examine the popularity of individual support activities across all districts, we 
calculated the proportion of our 2 1 districts that provided each activity. 

District Findings/Results 

In the following sections, we continue our discussion of policy alignment and CSR support with an 
examination of district-level approaches to CSR. First, we explore how the districts in our sample have 
worked to align and integrate CSR within their overall set of instructional and school improvement 
strategies. Next, we investigate patterns in district activities to support CSR, with a particular focus on 
district policies to mandate CSR for specific schools. We conclude with observations on the trajectory of 
CSR policy within these districts as well as the ways in which these districts refined or adjusted their 
approaches to CSR. 



*’ In cases where district respondents failed to answer all of the survey questions within a support category, support 
scores were determined by dividing the total number of “yes” responses by the total number of the survey items 
within the category that were actually answered. 
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Alignment of CSR With Other Improvement Strategies 

In an era of standards-based reform and accountability, districts face heightened pressure to ensure 
schools are meeting standards for student learning and achievement. Accordingly, they have employed a 
range of school improvement initiatives addressing such areas as curriculum and instruction, assessment, 
and professional development to facilitate schools’ progress toward meeting standards. Since 
comprehensive school reform by nature stretches across many of the areas in which districts are 
intervening, several opportunities exist for schools’ work with CSR models to conflict with strategies and 
priorities advanced by the district. 

To minimize the problem of competing initiatives at the school level, some districts have taken steps to 
situate CSR within their overall school improvement agenda. The chief academic officer in one district 
explained. 

What we’ve done is we’ve really done a great job of making sure that the schools and the 
model providers understand how what they do in the district supports the district’s goals, 
and they have really moved their schools forward with that. That has been a major thrust 
to say it’s not something that’s separate and apart from what the district is doing but it is, 
should be your way of actually supporting what the district is doing [Everville]. 

This section will examine how district administrators perceived CSR to fit in with other major initiatives 
and describe some of the steps districts took to align CSR with these initiatives. 

Alignment With Core Curriculum and Standards 

Although the districts in our sample varied in terms of the tactics and initiatives they employed to support 
school improvement, their approaches toward curriculum requirements pointed to two general patterns 
regarding districts’ strategies for reform. First, some districts centered their overall plan for school 
improvement around the implementation of a particular core curriculum or instructional model and 
typically focused district-provided professional development and other services on supporting teachers’ 
execution of that curriculum. Of the five districts in the qualitative component of this study, three fit this 
pattern of implementing a districtwide curricular program, and in each instance, district administrators 
stressed the importance of aligning CSR models with that program. 

To illustrate this curriculum-focused approach to reform, consider the mid-sized, urban school district 
Eastwicker. At the time of our study, Eastwicker was planning to roll out a locally developed curriculum 
based upon district content and performance standards. One administrator described the centrality of this 
new curriculum within the district’s plan for reform, declaring “The core of the system of course is the 
curriculum and . . . everything that we do should be aligned with that curriculum. Everything that we do 
in all of these different offices . . . should support it.” Because several schools were already implementing 
CSR models with specified curricula, the central office was in the process of collaborating with these 
schools to integrate their current CSR efforts into the upcoming district instructional model. Given the 
district’s commitment to having a uniform instructional model across schools, the district curriculum was 
to take precedence over CSR model curricula, though the alignment process was intended to emphasize 
commonalities between both. As one representative from the district articulated. 

We are working with principals and faculty to answer the question, how we could get 
Success for All, how we could get Voyager, how we can get all these other reform 
models that we’ve adopted to make sure that what they outline as the instructional 
strategy is aligned to the curriculum. That’s going to require some alignment of work; it 
has to be delivered. It has to be thoughtful. We’re not going to align our curriculum to 
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Success for All. Success for All will have to be aligned to our curriculum, and it can be 
because after all when one does an analysis, there’s a nice crossover of knowledge and 
skills. 

Officials from another urban district that recently adopted a districtwide curriculum expressed a similar 
emphasis on aligning CSR models with the district instructional strategy. Some indicated that CSR 
models’ compatibility with the district curriculum outweighed even the proven effectiveness of their 
components in influencing CSR decisions. The Chief Accountability Officer explained. 

I’m not concerned about the program. I’m not concerned about what works. I’m not 
concerned about that at all. I am concerned about alignment . . . The issue for me always 
is when you have a variety of reforms that you attempt to implement simultaneously . . . 
then the first casualty will be student achievement if you have misalignment [Riverton]. 

To ensure alignment at the school level, the district has instituted several measures during its schools’ 
CSR selection and evaluation processes. For schools applying for CSR funding, the district holds a 
technical assistance workshop to teach schools how to incorporate CSR into the district-mandated school 
improvement plan and how to promote alignment with other instructional initiatives like the district 
curriculum. When teams from the central office perform site visits to monitor schools’ progress with 
CSR, they use a protocol that includes a section regarding the CSR model’s alignment with the 
curriculum. Additionally, the district requires model developers to prove how their CSR model aligns 
with the curriculum. The district has also, on at least one occasion, sent its curriculum to a popular model 
developer working with its schools so that this developer could incorporate the district curriculum into its 
professional development workshops. 

The last of our three curriculum-mandating districts, Everville, had recently begun implementing the 
highly prescriptive Open Court reading curriculum and told a similar story of requiring CSR’s alignment 
with the district-chosen curriculum. Like Eastwicker and Riverton, this district prioritized its own 
curriculum selection; yet, rather than rejecting CSR as an incompatible strategy, it sought to integrate 
schools’ CSR work into their implementation of Open Court. The chief academic officer describes the 
district efforts to promote such integration as follows, 

I worked on the curriculum side of the house and pulled together all of the schools that 
had models and said to them that we were going to make sure that any school with a 
model focused the improvement effort on the Open Court and to see how the model, the 
components worked to support that improvement effort. So it didn’t matter if they were 
Comer, it didn’t matter if they were Different Ways of Knowing, anything. It had to wrap 
itself around the Open Court which was the district’s reading effort. Then, we charged the 
model provider with the same mandate, “Help the school make sure that tenets of the 
model were parallel in terms of making things happen with the Open Court.” 

Thus, as in Eastwicker and Riverton, officials in Everville opted to continue supporting CSR in 
conjunction with its new curriculum, provided that model developers and school stakeholders aligned 
their model with its curriculum to avoid interference. 



Other districts avoided mandating specific curricular materials and instead afforded schools wider 
discretion regarding instructional decision-making. Such districts may have recommended or restricted 
certain materials but generally supported a range of curriculum models for use in their schools. Yet, 
despite this flexibility with respect to curriculum, schools still needed to ensure that their approach to 
instruction fulfilled state and local content standards. An administrator from Oceanway, a district with no 
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required eurriculum but a large eoneentration of CSR sehools, emphasized the need for model developers 
to ealibrate their methods with state eontent requirements: 

You have to have some kind of eohesive alignment with what’s the state’s standard for 
what ehildren should know. That should be the eore. And the model developers need to 
find ways to lay their template on that. Beeause the bottom line is our kids don’t take 
their assessment, they take the state’s assessment. 

Thus, even though the distriet offered sehools latitude to implement CSR model-based eurrieula, it 
reeognized that, due to the demands of standards-based assessment, model developers had to adapt their 
strategies to eater to state standards. 

Integration With Other Reform Efforts: CSR as a Lever 

While distriets frequently eited the desire to prevent school-level conflicts as motivation for their work 
aligning CSR with other district reform strategies, several officials also noted the potential for CSR to 
serve as a lever for bolstering the implementation and effectiveness of other reform strategies. In such 
instances, CSR was not simply operating in tandem with another initiative but rather became integrated 
with that initiative or became a means by which to accomplish that initiative. 

As described in the previous section, administrators from Everville perceived CSR as a mechanism for 
“making things happen with Open Court.” They further articulated their vision of CSR as an “enabler” of 
reform, a strategy that ideally “facilitates and in no way detracts” from the district’s central Open Court 
initiative. The chief academic officer for this district described how CSR served as a means for preparing 
teachers to implement another schoolwide instructional program, “We found that what the models offered 
to them was an organizing framework to be able to fully implement the Open Court, and we find that 
those schools that had models were in a better state of readiness for collaborating on what they needed to 
do.” Another administrator elaborated on the ways in which CSR facilitated the implementation of Open 
Court, explaining. 

Those schools that have been doing a model are already organized to bring in Open 
Court. They’re already organized to bring in any curriculum initiative because they’re 
accustomed to working with each other because most models require that collaboration 
and teamwork and they’ve been working with them around that. You have extra hands in 
the school because part of what we do is we go and we listen to issues of just 
implementing change period and we help schools to think about that. 

The district thus established a connection between its central school improvement strategy and CSR 
which, by putting into place teacher professional community and support structures, laid the foundation 
for schools’ implementation of Open Court. Hence, in a policy environment characterized by a highly 
specified approach to instruction, districts can leverage comprehensive school reform’s broader 
capacity-building strategies to support other school improvement initiatives. 



In the context of a district with less specified requirements regarding instruction, CSR models can offer 
schools structures to fulfill more loosely defined district directives. Consider, for example, Dodgeland, a 
large urban district that grants schools primary decision-making authority over curriculum and textbook 
selection. When, as part of a districtwide reading initiative, the central office instituted mandatory blocks 
of instructional time reserved for reading, schools implementing literacy-focused CSR models could use 
the CSR materials and practices for reading instruction to comply with the district literacy block 
requirement. One observer from the district explained. 
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They’re mandated to have reading for two hours a day. There’s a general framework of 
four eategories that need to be touehed upon but unless the school is using something like 
an SFA, they are grasping at straws in terms of how do we meet this requirement and 
maximize that two hour block of time that they’re mandated to do as reading instruction. 

Accordingly, by providing low-capacity schools with tools for putting a district initiative in place, CSR 
can blend with other improvement efforts at the school level. 

Strategies to Integrate: Targeting Specific CSR Models 

In addition to facilitating the crossover of content between CSR models and other reform efforts, districts 
have established policies regarding schools’ adoption and selection of CSR models in order to promote 
school-level coherence. Everville restricted the types of CSR models that schools could choose in order to 
avoid models that might conflict with the upcoming implementation of the Open Court reading 
curriculum. According to one administrator. 

We decided as a district, rather than just have everybody doing everything, and they did 
make a decision as a district not to do the curriculum- [or] the reading-based models such 
as Success for All, Direct Instruction, because they were doing the reading series and 
didn’t want anything to compete with Open Court. The decision was made before the 
Open Court was decided upon, but they knew they were going to a reading program so 
they said they didn’t want something that would be a conflict. 

For schools already implementing highly scripted models like SFA, their model was either phased out or 
used only in an individual student workshop setting. In Dodgeland, the opposite was true: the district 
limited schools CSR model choices to models that did have a strong literacy focus so that they would 
facilitate the districtwide reading initiative. 

Riverton also promoted use of particular models to emphasize coherence with the district curriculum and 
other reform efforts. Though the district did not require schools to choose a particular CSR model, 
schools tended to select a model whose developer had worked closely with the district to align its 
instructional strategy with the district curriculum. The district viewed this model as “reinforcing what 
we’re trying to do here at the district” and presented it to schools as such. Furthermore, components of 
this CSR model tended to match with district initiatives such as school reading coaches, and schools were 
able to use their CSR funding to support a coach. 

Another good reason why people selected America’s Choice, they had to have a reading 
coach, a literacy coach and we recommended a teacher leader for reading. So that 
meshed. So they had, they could pay for half the person through America’s Choice and 
then they had money in their Title I grant to help reinforce the issue that they have their 
own literacy person in the building. 

Therefore, Riverton administrators perceived this model to be capable of not only minimizing clashes 
between CSR and the district curriculum but also of facilitating schools’ compliance with other district 
policies. By locating the model within the district policy context in this way, they likely contributed to 
this model’s prominence within the district. 
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Strategies to Integrate: Targeting Specific Schools for CSR 

Riverton further sought to prevent instances of school-level policy incoherence by targeting schools that 
were not already engaged in potentially conflicting reforms. One official explained how the district would 
consider declining federal CSR resources for schools already committed to other strategies. 

A variety of these schools that have these CSR grants are Corrective Action schools so 
from that perspective what we’re trying to ensure is that these resources, these supports 
and these programs are aligned to that school’s effort to improve but not competing with 
the district’s major reform efforts so that’s why sometimes, in certain instances, it’s better 
to say no to a set of resources if they’re going to be misaligned. 

Recognizing the need for schools to have the capacity to implement a CSR model before encouraging or 
requiring them to engage in this type of reform, the district reserved CSR for schools where it would have 
a greater likelihood of success and chose other improvement strategies for schools perceived to be less fit 
to implement. The district also used the federal CSR grant program to prioritize the use of CSR in schools 
that could most benefit from available CSR resources. One interviewee explained. 

What has happened in the past . . . our high poverty schools get googogs [sic] of money, 
even over a million dollars. And some schools that have very low poverty get very little 
money. So my boss’s idea was to offer these grants first to those schools that get little 
money for reform because schools that get a million dollars, they should be doing 
something worthwhile with that money ... So we try to match it with the needs of the 
school so that they could bring about a reform in their school program. 

By capitalizing on the availability of federal CSR money, the district treated CSR as a tool for financing 
reform in schools with less access to other resources. 

In order to pinpoint the schools they felt would benefit most from comprehensive school reform, some 
districts issued mandates compelling certain schools to adopt a model. Of the 2 1 districts in the overall 
NLECSR sample, 10 reported requiring at least some schools to implement a CSR model in 2002-03. 

In each of these 10 cases, the district’s CSR mandates were in some way associated with the district’s 
accountability system. Nine of these 1 0 districts required schools with a low accountability ranking, 
particularly those identified for improvement, to adopt CSR as a school improvement strategy. The 
remaining CSR-mandating district targeted schools that failed AYR, as did six of the other districts with 
CSR adoption requirements. By using accountability designations to target schools for comprehensive 
school reform, these districts positioned CSR as a tool within their overall accountability system, similar 
to the way in which many of the SEAs in our sample took steps to incorporate the CSR funding program 
into their overall accountability system. 

Supporting Schools During the CSR Process 

Districts developed approaches to CSR that fit their district culture and needs, and in doing so, some were 
more active than others. One way in which districts exercised control over the CSR process was through 
the adoption process, first by controlling which schools implemented CSR, and second by restricting the 
CSR models that schools could adopt. Another way in which districts sought to ensure that CSR met their 
needs was to provide a variety of supports that promoted full and smooth implementation of CSR. We 
found that districts that mandated CSR and districts that were more restrictive in model selection also 
tended to be more supportive of the CSR process in general. The following section will discuss support 
among all districts that mandate CSR and support among a subset of these districts — districts that 
mandate CSR and require schools to select a model from among a limited set. 
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District Mandates and Support 

NLECSR survey data indicate that when districts mandate the adoption of CSR models, they also tend to 
provide more support for CSR overall. When we plotted the district mandate variable against the total 
support variable, the resulting regression line yielded an r^ value of .21 . As we have seen, districts that 
mandate CSR often do so to address concerns over accountability and AYP. If districts are using CSR as 
a strategy to increase student achievement, they may have a vested interest in the success of the program, 
and therefore, offer more support. For example, all or nearly all mandating districts redesigned the budget 
system to allow for greater school autonomy, realigned district spending to support CSR,‘® and collected 
written reports from schools to monitor the implementation of CSR, whereas only between 46 and 54% of 
nonmandating districts provided the same supports. 

In particular, districts that mandate and require schools to choose from a limited set of models provide the 
highest overall support.^® Administrators may require that schools implement certain models, which they 
perceive to align well with district curricula or to address district- specific challenges, such as teacher and 
student mobility. These districts rank particularly high in funding support, community outreach support, 
networking support, and model selection support (see Figure 3.2). All of these most restrictive districts 
arranged meetings among faculty members whose schools shared a common reform model, a service 
offered by less than two thirds of the remaining districts. For example, one such district required monthly 
meetings for schools using each reform model. These districts also all provided public forums to discuss 
the CSR process with the community prior to model selection, an activity provided by just over half of the 
remaining districts. In Oceanway, for example, school management teams composed of community 
members, parents, and staff investigated and recommended models prior to adoption. There was also a 
willingness among these districts to facilitate negotiations with model developers, a move most likely 
prompted by a desire to ensure alignment between models and curricula. All but one of these districts 
communicated with model developers at least once per semester, compared to roughly half of all other 
districts. 

Districts that mandate CSR and require schools to choose from a certain set of models provide particular 
help in model selection that is rarely offered by other districts. Although these supports are infrequently 
provided by the whole sample, all but one of the districts that require schools to choose from a limited set 
of models provided one-on-one assistance in model selection, and all held model fairs. One district 
explained that the central office “presented the reform models to the schools.” Another district felt that 
their model fair promoted alignment to the district curriculum. A third district explained that, when they 
first started CSR, they specifically chose models that would not compete with the curriculum program 
they were implementing. 

On the other hand, none of the other districts that mandated CSR held model fairs. In one interesting case, 
the district administrator described how schools found models “despite a disinterest ... on the part of the 
central office.” Although the district mandated CSR for schools identified as low achievers, “there’s never 
been a push to share those models, to encourage schools to adopt a model and so schools are, have 
basically been totally on their own in terms of their initiative and finding examples of models.” This 
district was also the only mandating district that did not provide funds for visiting CSR schools in action. 



There were three exeeptions: two distriets (Arborwood and Dodgeland) that mandated CSR but had lower support, 
and one (Gableton) that did not mandate CSR but had high support. 

One mandating distriet did not provide this support. 

The five distriets that required sehools to seleet from a limited set of models were among the six distriets that 
ranked the highest in overall support. 
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Figure 3.2. Support in mandating districts that require schools to select from a set of models 
vs. support in all other districts 




District Support Activities 

Survey data suggest that districts provided a wide array of support activities. Among the CSR support 
strategies measured by the NLECSR district survey, districts reported providing from 50 to 93% of the 
different types of support, and the average level of support was 74% (see Table 3.2). 

While districts were highly supportive overall, some support activities occurred more frequently than 
others. One hundred percent of districts^' assisted schools in securing resources for CSR. Just over 95% 
of districts had specific funds for CSR and provided instructional leadership throughout the 
implementation of CSR. A similar proportion of districts engaged in efforts to involve the community in 
the reform process. Districts had various strategies to involve the community, including local school 
councils, community improvement teams, school site meetings, and town hall meetings. Other districts 
included parents on their school improvement teams. Ninety-five percent of districts also sent their staff 
onsite visits to monitor implementation; however, the frequency of these visits ranged from once a week 
to once a year. Respondents reported that site visits focused on level and fidelity of CSR implementation, 
student achievement data, classroom performance, professional development offerings, instructional 
strategies, and alignment with content standards. 

In contrast, some supports were provided infrequently (see Figure 3.3). Only 29% of districts held model 
fairs and 48% provided one-on-one help to schools during model selection. Districts that held model fairs 
reported that they were widely attended by district and school administration, teachers, parents, 
community representatives, and sometimes students. Districts that provided one-on-one consultation 
during model selection usually did so upon request or as needed by the school. Assistance focused on 
needs assessments, researching models, securing external partners, writing grants, identifying best 
practices, working on budgets and contracts, and locating schools to visit that were implementing models. 
With regard to the implementation process, only 32% of districts increased professional development 
days, and one third increased common planning time to better support CSR models. While over half of 
district communicated with model developers, only 38% met with developers at least once per semester. 



21 



All percentages are based on the number of districts that answered the item and may not be equal to the total of 21 
districts. 
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Table 3.2. Districts' total support scores 



District 


Percentage of total support 
activities provided 


Everville 


50 


Rainfield 


55 


Arborwood 


62 


Mount Violet 


66 


Roseburg 


69 


Huntertown 


69 


Dodgeland 


69 


Lilyport 


71 


Hickoryville 


72 


Meridian 


72 


Westlake 


72 


Kentland 


74 


Jasmineton 


76 


Oak Valley 


79 


Riverton 


79 


Pineford 


79 


Gableton 


83 


Elm County 


83 


Eastwicker 


90 


Oceanway 


90 


Greenbower 


93 



Figure 3.3. Support activities provided most often and least often 
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Altering and Refining District Strategies for CSR 

The Future of CSR 

CSR policies are shaped in an ever-changing atmosphere that is unique to each district. With high 
pressure for student achievement, districts have moved to change or discard initiatives that are not quickly 
perceived as effective in raising student test scores. Faced with this pressure to find the solution to student 
achievement challenges, districts are evaluating the extent to which CSR meets their needs. Among the 
NLECSR case study districts, we found that one district was refining its approach for the continuation of 
CSR. In this district, CSR was an active process, with the district refining the reform methods to increase 
efficacy and promote sustainability. A second district was undecided as to whether to continue with CSR, 
and was in the midst of investigating best practices from the reform models. The remaining three districts 
were phasing out CSR. 

Refining CSR 

As districts implemented CSR, many engaged in policy refinement as they learned from their successes 
and mistakes, and aimed to make CSR work at the district level. One such district from this study was 
Riverton, which learned from their initial interactions with reform model developers and used that 
knowledge to refine their approach to later negotiations. After finding that some model developers failed 
to deliver on their promised support, the district learned to “have everything in writing.” District 
personnel noticed that “when we first started, I think they all decided they would see what they could get 
away with, like not showing up for staff development or saying they were gonna send five people and one 
person showing up,” but once the model developers realized that the district “[kept] a tab on those 
things,” they began to follow through on their contracts. The district also became more firm in the 
negotiation of the contracts. As the district administrator told principals, the model developers “don’t tell 
you what to do; you mutually agree. . .this is our money and we will use it the way the school best sees fit, 
not the way this individual developer thinks it should be used.” The administrator cited one particular 
example in which the district backed the school in refusing to pay the cost of a year-long contract when 
the work period was shortened to 6 months because of a funding delay. 

Perhaps because this district has worked to improve and refine their approach to CSR, they are working to 
sustain reform, even as grant cycles conclude. Indeed, the district has “urged schools, after the three years 
[of the CSR grant] have finished, ... to continue,” and many schools have used their own funds to do so. 
Although some schools have not maintained their affiliation with the CSR model developer, district 
administrators have observed that many CSR practices have been integrated and institutionalized at the 
school level. 

Indecision Regarding CSR 

By 2002-03, one case study district was undecided on the fate of CSR. One district administrator 
described the status of CSR in Dodgeland: 

[W]e’re looking at it closely ... I think what we’ve tried to do is really be thoughtful in 
what we’re doing and look critically at data. Look at what has worked successfully in 
other districts and really build programs that incorporate those features. If we find 
something’s not working we [say], “Let’s get out of this one.” 

The administrator further described the district’s approach to evaluation: in the cases of models that have 
not had an impact, “we hope to be able to gather enough information to really understand why and if it’s 
something that can be reconciled and whether or not we need to look in other directions for other 
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supports.” An example of this strategy ean be seen in a reform model that schools stopped using because 
it was not meeting their needs, but which has returned to the district after making some changes to 
address the schools’ concerns. Although the district said it would stop using models that it decides are not 
working for its schools, it appears that district officials have not yet ruled out CSR altogether. One official 
thought that “there are probably situations where we may actually adopt [a model] for some of our lowest 
performing schools who don’t have the internal capacity to support the rigorous work that we want 
occurring.” However, the model must be “appropriate” and “aligned with our new district priorities.” 

Phasing out CSR 

Finally, three districts were already phasing out reform models. One case study district described their 
efforts to “streamline our program ... to rid ourselves of those [efforts] that are not proving themselves.” 
Another district likened their approach to refining district reform as a housecleaning: “you end up with 
really a very cluttered school program ... so there was a real effort to clear some of, to clear all of it out 
quite frankly, and a lot of CSRDs went with it.” According to this district administrator, 75% of schools 
operating CSR models through CSRD grants dropped the models by 2002-03. 

District administrators provided several reasons as to why schools discontinued their CSR models. Both 
teacher turnover, which presented challenges for training and program buy-in, and administration 
changes, wherein the new principal does not support the model, were cited as reasons that models were 
phased out. In fact, two district officials attributed the success or failure of models to “how the staff is 
working in concert . . . the dynamics, the stability of the school, the buy-in of the staff, the ability of the 
leadership to really engage them” rather than the characteristics of the models themselves. Schools also 
complained that they had to spend too much time on “administrative paper, business time with [district] 
staff, which drains them of time and energy.” 

Another key concern, augmented by NCLB accountability, was pressure to quickly produce results; as 
one administrator said, “either it is something which can be used to help kids’ achievement this year or 
forget it.” In one focus study district in particular, administrators and educators perceived that the most 
frequently adopted models were poorly aligned with the main accountability test, and because of this, 
schools failed to produce expected gains. 



Connecting State and District Strategies 



As states and districts each developed strategies to assist schools’ implementation of CSR, policymakers 
at both levels encountered similar issues related to promoting policy coherence, providing useful support, 
and refining policies to better suit school, district, and state needs. Often, state and district roles 
intersected, as states urged districts to recognize and support schools, and as districts benefited from the 
funding and technical assistance that states provided. The intersection of these roles led to different 
state-district dynamics: in some cases, states and districts reported similar levels of engagement with CSR 
while in other cases one level seemed to compensate for the other. Some states and districts made efforts 
to collaborate and to align their CSR strategies to best support school needs, but others encountered 
situations where state and district CSR policies were at odds with one another. This section will explore 
how selected state-district pairs from the NLECSR study interacted with regard to policy integration, 
support, and refinement. 



Alignment and Integration 




State and district officials typically expressed similar concern over minimizing any interference resulting 
from schools’ engagement in multiple reform efforts. Recall, for example, Everville, where the 
districtwide implementation of Open Court stimulated efforts to align all CSR models active in the district 
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to fit with the new eurrieulum. Administrators in Everville promoted the eontinued use of CSR models as 
a means of faeilitating the Open Court initiative, hoping the two strategies would work in unison to 
eonstitute sehools’ means of improvement. The view that CSR models should serve as one integrated 
eomponent of schools’ total reform efforts echoes the vision espoused by Everville ’s SEA. The CSR 
coordinator for this state explained, “When we orient schools on that first occasion after they get a [CSR 
grant] award, we make it very clear to them that they should have one school improvement process and it 
ought to be named comprehensive school reform and then any other initiatives that are going on in the 
district would come under the umbrella of comprehensive school reform.” 

Furthermore, both state and district officials advised against implementing curriculum-centered CSR 
models in schools using highly prescriptive curriculums such as Open Court, which might conflict with 
the CSR model’s instructional plan. To avoid such conflicts, the district explicitly restricted schools’ 
selection of curriculum-based models that were not aligned with Open Court. At the state level, the CSR 
coordinator described attempts to prevent schools from encountering discordant curricula as a result of 
their participation in multiple federal programs: 

It would be a problem to impose a Reading First grant on top of a CSR grant — not that 
we don’t have a few schools doing it because we do — but it has to be pretty artfully done. 

It would be a mistake, for instance, to have an SEA implementation and an Open Court 
implementation going on in the same building to satisfy CSR and Reading First. We have 
an obligation internally and we do have these discussions periodically about that point at 
which programs interface and how we manage that effectively. 

Thus, in the interest of promoting instructional coherence in Everville schools, officials at both the state 
and district level placed limitations on schools’ adoption of potentially competing instructional strategies. 

In addition to emphasizing CSR models’ accord with other school improvement and instructional 
processes, both state- and district-level administrators sought to ensure unencumbered implementation of 
CSR by targeting particular schools in which CSR was perceived to be the most effective strategy. The 
state CSR coordinator responsible for Everville described efforts to engage districts in the CSR process to 
prevent them from imposing additional misaligned improvement initiatives. He explained, “We want the 
district to have an awareness [of schools’ CSR implementations] because many of them lose sight of the 
fact that schools in their district already have a school improvement process going on and that they have 
to honor that and if they fail to honor it, which we’ve had happen on occasion, we consider pulling 
funding.” 

Everville ’s district officials acknowledged the tension between their desire to have schools continue with 
CSR to maintain the often sizable CSR grants and their desire to see schools engaging in what they 
perceive to be the most appropriate type of reform. One district administrator noted this concern, 
particularly as it related to the district’s “housecleaning” of ineffective CSR programs: “I know there was 
a lot of discussion [about policy alignment] among the state when we were training ... I think it assisted 
in a compromise where ... if the school can show that they have been progressing, achieving and they 
have a model, go ahead.” Thus, in determining how CSR fit into their strategy for school improvement, 
the district weighed the option of receiving state CSR funding and technical support against the perceived 
utility of continuing with CSR implementations in particular schools. 

Support 

States and districts provided various means of support for the CSR process to districts and schools in their 
jurisdictions. Our analyses ranked both states and districts to determine those that provided the most 
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support activities and those that provided the least. Sometimes, a state and its districts provided similar 
levels of support, whether it was a high level of support, a low level of support, or somewhere in between. 
Other states were at odds with their districts, with one education agency providing a high level of support 
while the other provided very little. This section will discuss the types of support provided in two 
state-district pairs representing different patterns of support. 

Some states and districts reported similar levels of engagement with CSR, whether relatively low or fairly 
high. An example of the latter is Riverton and its SEA. Both the state- and district-level agencies were 
among the most supportive in our sample. The state focused much of its assistance on the activities 
preceding the disbursement of CSR grants, beginning with a series of mandatory workshops for schools 
and districts interested in CSR. After schools attended the workshops, according to one State official, “the 
extent of their assistance is really up to them. Some schools ask for a lot of assistance in putting their RFP 
together and putting their needs assessment together, and others you don’t hear from.” One requirement 
of the proposal process was to conduct a needs assessment to use in making decisions about reform. The 
state held “a lot of workshops on putting together comprehensive needs assessments, ensuring that 
schools are taking at least 6-8 months pulling this information together ... so that when they start making 
decisions. . .they’re making them based on the most data that they can possibly have . . .” 

Likewise, the district of Riverton also reported assisting schools in conducting a formal needs assessment. 
This assistance, however, came not in the form of workshops, but through site visits, a review of state 
testing programs, and help during the school improvement planning process. Thus, the state and district 
provided complementary strategies to provide comprehensive assistance in conducting a needs 
assessment. Schools could also request individualized assistance from the state CSR program manager 
during the grant-writing process. She said, “some schools, more this year than ever, have asked me to 
look at different sections . . . and see if they have the right information.” The high degree of state 
assistance seemed to be effective, as the CSR program manager was “happy to say that [schools] that 
actually do ask for the most assistance in all cases have gotten their grants approved.” 

Pineford and its SEA present a different example of state and district support. Pineford was among the 
most highly supportive districts, based on survey data, while its SEA was one of the least supportive. It 
appears as if the district of Pineford compensated for the lack of support from the state by offering a high 
level of district support for CSR. For example, the state offered minimal assistance during the grant- 
writing process, and guidance on selecting CSR models was sparse as well. The state simply advised 
districts and schools to select a model based on scientifically based research and that follows the 1 1 
components of CSR. The district, however, provided further assistance to its schools. Pineford assisted its 
schools in conducting needs assessments by providing the schools with workshops, training, and data. 

The district provided funds for teachers, administrators, specialists, and parents to travel to schools 
already implementing models in order to see CSR in action. Finally, the district also provided written 
documentation on different CSR models and helped schools to identify the models that were aligned with 
district standards and assessments. 

Pineford also rated high on our survey index of support for implementation. In contrast, at the state level, 
there was only one person working on CSR, and state support for implementation was provided 
essentially “when they ask for it.” The district, on the other hand, provided a great deal of support during 
implementation. Pineford communicated and met with model developers at least once per semester and 
facilitated negotiations with models. They facilitated meetings between principals and faculties in schools 
using the same model. The district provided instructional leadership to teachers throughout 
implementation, and increased the amount of common planning time for teachers to better support CSR. 
Finally, Pineford required that school improvement plans focus on design implementation. The state 
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supervisor also remarked, “we encourage [the districts] to support [CSR] ... the district needs to pitch in 
for substitutes or . . . funds.” And Pineford did so, providing 83% of funding supports, including 
providing district funds for CSR schools, assisting schools in securing resources for CSR, and realigning 
district spending levels and patterns to better support CSR. Overall, it appeared that Pineford’ s high level 
of support for CSR compensated for the low level of support from the state. 

Refinement 

As we have seen, districts and states have had different trajectories in their evolving CSR policies and 
practices. In some states, CSR policy stagnated; in others, it underwent a dramatic change. Many districts, 
at the time of our study, were uncertain of the future of CSR or were phasing it out. Still other states and 
districts engaged in continual refinement of CSR. Policy changes within a state are likely to influence 
policy or practice in districts, and both states and districts placed an emphasis on similar themes during 
the implementation of CSR. This section will explore the connections between state and district level 
changes to CSR policies and practices, specifically within one state-district pair, which continuously 
refined their CSR process. 

Both the Riverton district and its SEA remained open to refining their implementation methods in order to 
ensure they were best meeting state, district, and school needs. As described earlier, this state altered the 
focus of their model fairs because they felt that a focus on the models as commercial packages 
encouraged schools to think that they could buy something that would fix all their problems. Their new 
“strategy for design fairs is that we just have people talk about the kinds of things that work without really 
buying into a specific model.” This new focus on reform strategies forced schools to think about what 
they would be putting in place. Riverton, too, learned from early mistakes. District respondents became 
more firm with model developers when they found that the developers did not always follow through on 
their promises. 

Likewise, both state and district officials described a parallel emphasis on needs assessments and 
data-based decision making. As the state official explained, “We started to focus less on what were the 
strategies that were going to be put in place and more on the needs assessments, finding out what exactly 
it is that doesn’t work in your school.” The district also stressed similar issues. District officials told 
schools that, when negotiating with model developers, they should focus on “what are [the school’s] 
needs and how are [the model developers] gonna fulfill those needs.” Although the district recognized 
how important it was to conduct needs assessments and use data when planning school reforms, officials 
felt that the schools were lacking the capacity to analyze the data. One official stated, “While schools may 
be aware of the data that they have, they have no systemic way to approach it.” Perhaps as a result, the 
state began to offer “a lot of workshops on putting together comprehensive needs assessments” to aid 
schools in data-based decision making. 

State actions can affect operations at the district and school level. When the SEA started encouraging 
school districts to put forth larger grant requests, Riverton and its schools responded. The state CSR 
program manager “encouraged [districts and schools] to look really realistically at their budget . . . and 
not to cut themselves short.” Many districts had been requesting the minimum of $50,000, thinking that 
larger grants would not get funded. The state wanted districts to consider the real cost of the reform they 
were trying to implement. Riverton explained that the schools that received the earlier CSR grants were 
funded at the minimum, but the high schools who received later grants “got big chunks of money . . . 
because they said they needed more money to bring about some worthwhile reform and you can’t do it 
with $50,000.” Thus the districts’ change in funding strategy seems to be a result of the clarification of 
policy at the state level. 
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Discussion 

School districts and SEAs can have a potentially important role in shaping the suceess — or demise — of 
CSR at the school level. While it is important for states and districts to foster an administrative context 
that facilitates CSR, the NLECSR data suggest that it is more critical that they limit conditions that would 
inhibit successful implementation of CSR. 

Indeed, there are many state- and district-level conditions that can make CSR implementation even more 
challenging. Among these is the proliferation of competing (and distracting) reform strategies which 
divert school staff from focusing on CSR. This is particularly problematic in high-poverty, low- 
performing schools that may be eligible for numerous improvement grants but lack the capacity to 
carefully select reform strategies. Hence, a primary tactic among state and district CSR administrators in 
the NEECSR sample was to seek coherence and alignment among CSR and other approaches to school 
improvement, including those that were districtwide (such as curriculum) and those at the school level 
(such as Reading First initiatives). 

As we have seen, both states and districts promote school level coherence by targeting which schools 
were to receive CSR grants. Some states and districts crafted eligibility policies that limited the 
participation of schools that may have initiatives or curriculums that would conflict with CSR. In some 
cases, both states and districts looked for schools with the capacity to implement large-scale reform or 
schools that had not had many opportunities to receive federal grant money. At the district level, another 
strategy for alignment was through the selection of CSR models that would align with or even promote 
other district initiatives (and the restriction of models that would not align). 

SEAs operate at the macro level of the educational process, yet state actions shape schools and districts. 
Because states are the conduit for many federal grants, a proactive state approach can help to ensure that 
these grant monies — and the ensuing efforts by school staff — are not competing. Moreover, states have 
increasing obligations to support schools that are in the latest stages of accountability designations (most 
notably those in restructuring status under NCEB) and their level of engagement in the school 
improvement process has been augmented over the past several years. 

The prior literature on district support highlights some key activities that may foster higher levels of CSR 
implementation. However, among the districts in the NEECSR sample, it seems that few had developed a 
focused set of support strategies. Rather, districts generally engaged in most of the 29 support strategies 
about which the district survey asked. One of the key differences appeared to be among a subset of 
districts — ^those that required schools to select from a limited set of models — which were more proactive 
with regard to specific supports, including assistance during the model selection process. State support 
tended to focus more on grant writing then on implementation support, and, like districts, the level of 
support varied. Some states and their districts offered similar levels of support, indicating perhaps the 
prominence, or lack thereof, of CSR. In other cases, it seems that highly supportive districts may be 
compensating for a lack of support from the state, or vice versa. 

Finally, both states and districts can influence school reform activities through rhetoric and public 
position statements. That is, they may communicate in subtle or obvious ways that particular reform 
strategies have fallen out of favor, thus discouraging schools from vigorously pursuing CSR. Among 
these are “housekeeping” districts that are attempting to “clean out” CSR, or states that administer federal 
grants with the barest required effort. In contrast, states and districts with a viable feedback loop — ^those 
that actively engage in policy refinement — are most likely to successfully communicate an ongoing 
commitment to reform, and to develop strategies that will encourage long-term sustainability of CSR. 
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In exploring state and distriet strategies for integration and support of CSR, we have sought to establish 
the context of states and districts implementing CSR. While the rest of this report focuses primarily on the 
life cycle of CSR at the school level, it is nonetheless important to consider the state and district strategies 
that can influence how the stages of the life cycle unfold. For example, state and district support activities 
to inform schools about CSR opportunities impact the adoption process in individual schools. State and 
district resources can provide assistance throughout the implementation process at the school level. 
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Introduction 

Teachers are an essential component to the ultimate success of school reform; it is they who must commit 
to the hard work of educational change. For teachers to become substantively engaged in implementation, 
scholars suggest that they must first “buy into” the general premise of reform (Datnow & Castellano, 
2000). They must accept the general premise that reform is important and that the proposed reform in 
particular is appropriate for the school. Scholars of school reform assert that an important mechanism 
through which schools can ensure teacher buy-in is the process through which a school selects a reform 
strategy (in this chapter, referred to as the “adoption process”), including early consideration of teachers’ 
beliefs and values (Weiss, 1995; Eisenhart, Cuthbert, Shrum, & Flarding, 1998; Bodilly, 1998; Ross, 
Alberg, & Nunnery, 1999). That is, if teachers are engaged in and supportive of the decision to adopt the 
model, they will be more likely to approach implementation with a positive attitude. 

This report is grounded in the notion that the metaphor of a life cycle is informative for understanding 
processes associated with comprehensive school reform. Within this metaphor, the adoption of a 
comprehensive school reform model is the initiating phase — or the birth — of reform. The initiating phase 
consists of the decision to undertake reform and the selection of a specific CSR model that will address 
school needs. This chapter addresses the earliest phases of reform, and examines the extent to which the 
“birth” of reform shapes its later success. 

This chapter seeks to address part of the NLECSR Research Question 3: What supporting conditions and 
strategies are necessary to effectively implement and sustain reform in schools? The analyses integrate 
both qualitative and quantitative data to determine the extent to which the quality of the adoption process 
is indeed a necessary precursor to effective implementation. 

The organization of this chapter is as follows. First, we briefly review the literature on the model adoption 
process, focusing on the constructs that most frequently characterize the quality of the adoption process. 
Next, we present a theoretical framework for understanding the quality and potential implications of the 
CSR adoption process. Third, we describe the adoption process from a qualitative perspective, using data 
coded through an analytic rubric developed specific for the NEECSR study. We then turn to survey data 
to describe the prevalence of specific adoption processes, and to identify relationships between voting and 
constructs related to implementation. We conclude with a revised view of the adoption process and policy 
recommendations based on our analysis of the model adoption process in schools. 

Literature Review and Conceptual Framework 

The quality of the adoption process has potentially significant implications for the later success of a CSR 
model, as reflected by the research literature on this topic. In reviewing this literature, we will distill the 
dimensions that most frequently reflect the quality of the model adoption process, as well as the main 
phases of this process. 
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Three eore dimensions most often charaeterize the quality of the adoption proeess: the degree to whieh 
the proeess is informative (providing adequate information to stakeholders), inclusive (enabling 
stakeholders to have a voice in selecting a model), and legitimate (ensuring that stakeholders’ voice is 
uninhibited and unconstrained by external pressures). In one of the earliest and most comprehensive 
studies of the model selection process, Bodilly (1998) determined that “schools were likely to make more 
significant implementation progress within the two-year time frame studied if they: Were well informed 
about the designs [and] had free choice among designs” (p. 56). These concepts of information and free 
choice emerge in other studies, albeit with alternate terminology. For example, Datnow (2005) concludes 
that a more “participatory” adoption process is more likely to foster enthusiasm and support for 
implementation (p.l 1). Writing of the Success for All model, Slavin and Madden (1999) note, “We have 
found that it is very difficult to work in schools in which the staff did not make an overwhelming, 
informed, and unfettered choice” (p. 8). 

However, much of this same literature concludes that the model adoption process is frequently lacking in 
these key dimensions. A report by the Education Commission of the States (1999) puts it plainly: 
“Although it seems obvious, many administrators, policymakers and others advocating reform fail to 
bring teachers to the table early on. . . . Without the active support of a majority of teachers, 
comprehensive reform is doomed” (p. 16). 

Indeed, numerous studies have highlighted deficiencies of the model adoption process. Teachers in 
reforming schools generally lack time to become engaged in the adoption process and do not feel 
informed about the models (Berends & Bodilly, 1998; Stringfield & Ross, 1997). Ross (2001) noted that 
teachers and principals in Memphis, TN, often expressed the belief that the model adoption process was 
not entirely as flexible or democratic as the district had thought it would be: “Many conveyed the belief 
that the external models (especially a select few) were ‘favored’ by the central administration. . . . [They] 
reported that such models were often hastily selected by schools without a true understanding of their 
focus” (p. 7). Other studies have reported that teachers were unsure of why their school had decided to 
undertake reform, selected a reform with little research, or engaged in a voting process that failed to foster 
readiness for reform (Millsap, Chase, Obeidallah, Perez-Smith, Brigham, & Johnston, 2000; Smith et al., 
1998; Smith et al., 1997). 

Datnow (2000) conducted a qualitative analysis of the adoption process in a small sample of schools, 
highlighting some of the dynamics of this process. First, she found distinct patterns with regard to the 
impetus for reform: In some cases, the district obliged schools to adopt a specific model, and in others, 
the district encouraged schools to adopt one of several approved models. In still other sites, the adoption 
of the model was initiated by school stakeholders themselves, with no intervention from the district. In 
schools in which the adoption process was largely driven by the principal, teachers played a more 
substantive role — although these reports were not unanimous. Moreover, Datnow reported that 5 of the 7 
schools with principal-led reform were still implementing their models. In schools in which the district 
played a major role, teachers were required to vote for the adoption of the model; despite this, “local 
buy-in was not as genuine as was hoped, despite the best intentions of district administrators.” In some 
cases, it appeared that the voting process either lacked salience for teachers (some did not remember 
voting) or was mildly coercive. As Datnow explains. 

In many cases, teachers stated that the reform adoption voting process was not genuine. 

Either teachers voted several times until the desired outcome was achieved, or they were 
strongly encouraged to vote for the reform the first time. . . . The vote gives the 
presumption of buy-in, and allows administrators to later point to the fact that staff chose 
to adopt the reform, (p. 367) 
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Drawing from the themes in this literature — ineluding descriptions of perceived problems — we can 
suggest a conceptual model of the type of adoption process that is most likely to foster implementation 
(see Figure 4.1). One key feature of the adoption process is the degree to which it is informative — that is, 
the extent to which stakeholders receive access to information on the models the school is considering. 
Schools’ efforts to provide information might be cursory — for example, teachers might listen to short 
presentations about models during a single faculty meeting. In contrast, other schools engage in a more 
substantive approach, sending teachers to visit schools implementing models or inviting model 
representatives to respond directly to faculty questions. In the “ideal” scenario, the information-gathering 
process should also be inclusive, that is, all stakeholders should have the opportunity to learn about the 
proposed models. 

Figure 4.1. General conceptual model of the adoption process 




The next key feature — as reflected by previous research — is the extent to which model adoption is 
inclusive, particularly the way in which stakeholders share their opinions with regard to which model is 
ultimately adopted. Teachers may express their views through different processes, most frequently a vote 
or a consensus-building process. In either case, scholars suggest that the process should be inclusive; most 
or all stakeholders should be afforded an opportunity to voice their opinions in an environment that does 
not stifle dissent. In the case of many models, the participation of teachers in the decision-making 
process — in most cases, voting for the model — has become a mandated precursor to implementation. The 
assumption on the part of model developers is that a high percentage of teachers voting in favor of the 
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model is an indicator of buy-in, and that “the vote itself may create the conditions for coherent 
implementation” (Tushnet, Flaherty, & Smith, 2004, p. 37). To best support implementation, the literature 
suggests that this process have legitimacy, or be “unfettered;” it should be more than a perfunctory 
approval of a forgone conclusion. 

The current literature suggests that an adoption process that is inclusive, informative, and legitimate is 
most likely to ensure that teachers have adequate understanding of the model and will generate teacher 
buy-in (favorable disposition toward the model). These features interact with the existing school culture, 
most notably principal leadership and the professional community that exists among teachers. 

This chapter addresses several specific subquestions, within the context of this conceptual framework. 
First, how prevalent are specific model adoption practices? Do adoption processes appear to vary by 
school and/or district context? Next, does the level of CSR implementation appear to vary based on the 
adoption process? And finally, when schools select and adopt CSR models, are these processes 
informative, inclusive, and legitimate? That is, do they include dimensions that are most frequently 
associated with higher quality adoption processes? 

Methods/Analytical Approach 
Data 

Qualitative Data Sources 

Analyses related to the model adoption process drew from qualitative data, most notably from site visits 
to 22 schools in five districts. (While the full case study sample included 32 schools, 10 of these did not 
yield data on the adoption process.) All 22 schools were visited during the first wave of data collection 
(the 2002-03 school year) and 1 0 schools received a follow-up visit during the 2003-04 school year. The 
primary data sources for the adoption analyses included teacher interviews, teacher focus groups, 
principal interviews, and interviews with reform facilitators. All interviews were recorded and transcribed 
for coding. In all, over 200 interviews or focus group transcripts were analyzed for the analysis of the 
model adoption process. 

Quantitative Data Sources 

The quantitative component consisted primarily of surveys of teachers, principals, and district 
administrators, in both schools with CSR models and matched comparison schools. The surveys of 
teachers were administered during the 2001-02 and 2003-04 school years, whereas the principal and 
district surveys were administered for 3 consecutive years, starting in 2001-02. Selected questions from 
all three surveys were used in analyses of the model adoption process. However, analyses that 
investigated the relationship between adoption and implementation used only the 2002 principal data set, 
which provided most information regarding the adoption process of CSR programs. Two hundred six 
principals reported whether a school voted for a CSR model and the percentage of faculty voting for the 
adoption of a specific model. 

In addition, implementation analyses included only schools that were known to implement a CSR 
program; hence all comparison schools were excluded from adoption analyses. Moreover, focused 
analyses related to model implementation could include only schools that were implementing a CSR 
program for which we had the dependent variable. Level of Total Implementation. Level of Total 
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Implementation was an index that eombined all different implementation indices (12) to one aggregated 
measure of level of implementation.^^ 

Analytic Models and Methods 

Qualitative Analysis Based on the NLECSR Analytic Rubric 

To ensure the analysis of qualitative data was as systematic, balanced, and rigorous as possible, we 
developed an analytic rubric comprised of key elements related to the CSR reform process. The rubric 
was designed to facilitate the analysis of school-level support for comprehensive school reform, and 
includes constructs that are frequently associated with implementation. The rubric included constructs 
related to the following: 

♦ Stakeholder understanding of the CSR model 

■ Comprehensiveness (the degree to which stakeholders perceived the model to be 
multidimensional) 

■ Consistency (the degree to which stakeholders perceived the model to be used on a 
schoolwide basis) 

♦ Stakeholder perceptions of the CSR model 

■ Perceived quality of the model 

■ Perceived appropriateness of the model 

■ Challenges associated with the model 

♦ School-level CSR processes 

■ Model adoption process 

■ Quality of developer supports 

♦ Professional resources 

■ Quality of principal leadership 

■ Presence of professional community for reform 

■ Quality of district leadership 

Each construct was divided into five descriptive levels to be used in coding. For example, with regard to 
presence of professional community for reform, level 4 (the highest) read, “Interviewees describe a 
professional community in which there is trust, communication, and exchange of ideas,” while the lowest 
rating level read, “There is little or no professional engagement or community; commentary is negative.” 
Two rubric components were used in the analyses for this chapter: model adoption process, and 
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Comprehensive details on the way in whieh Level of Total Implementation was determined are provided in 
chapter 5 of this report. 
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comprehensiveness. More complete rubric examples for each of these are presented in Table 4.1; the 
entire rubric is included in appendix A. 

To complete the rubric, an analyst would review all interview and focus group data for each school site, 
coding data that pertained to each construct. For each interview that included data related to a construct in 
the rubric, the analyst would assign a numeric code and would document a quotation that provided 
evidence for the coding. With all data coded, we were able to generate an aggregate score for each CSR 
school, both for individual constructs (such as model adoption) as well as a composite score for all 
constructs related to the CSR model. The composite score was calculated by first generating average 
scores from three groups: (1) teacher interviews; (2) teacher focus groups; and (3) principal, assistant 
principal, and reform coordinator interviews. Because of the pace, timing, and flow of some interviews, 
there were constructs for which some interviews did not provide adequate data. For these constructs, the 
analyst would indicate “ND” (no data). Because we did not want to allocate undue weight to data 
provided by very few teachers, we only averaged teacher scores if were able to glean data from at least 
half of the interviewed teachers. If an insufficient number of respondents provided data on a specific 
construct, we did not attempt to generate an overall score. The final score for each construct was 
calculated as a percentage of the total possible scale, so that all constructs were placed on a scale 
ofO-100. 

Analytic rubrics were completed for all schools from which the team had collected case study data. Two 
analysts conducted the bulk of the rubric coding. For each school, the analysts discussed the coding in 
depth, coming to a consensus for each score, and calibrating the scores of all constructs. 

A final step helped ensure the validity and reliability of the rubric ratings. In a separate analytic process, 
researchers reviewed all coded interview data from the case study schools, and prepared detailed case 
study summaries. Each school profile followed a common template, which included sections on the model 
adoption process. Rubric ratings for core constructs — including the adoption process and teacher 
perceptions of model comprehensiveness — were checked against the school profiles. When the team 
perceived minor discrepancies between each, the ratings were discussed and adjusted, as appropriate. This 
intensive review process ensured that the ratings reflect, as accurately as possible, the range of 
implementation dynamics that occurred within the case study schools. 

With completed rubric data, several analyses of quality of the adoption process were possible. These 
included analyses of the overall variation across the sample, clusters within the sample, variation by 
district, variation by model type, and variation in relation to other constructs. 

Quantitative Analyses Based on NLECSR Survey Data 

While the qualitative data enabled a systematic analysis of the adoption processes within a small sample 
of case study schools, the intent of the survey analyses was to determine the extent to which certain 
practices associated with the adoption process (most notably, voting to adopt a model) have become 
institutionalized. In addition, we sought to determine whether adoption practices (again, voting) were 
associated with survey scales that bode favorably for implementation. These included scales related to 
professional community, clear and shared goals, and professional development, among others. 
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Table 4.1. Excerpt from NLECSR analytic rubric 





Construct 




Teacher Ratings 


Construct Name 


Description 


Descriptors of Levels 


T1 


T2 


T3 


Quotes 


Adoption process 


This construct 
reflects the degree 
to which 


4: The teacher or principal describes an adoptien process 
that was inclusive and informative, and stakehelders had an 
opportunity to express their views. 












stakeholders 
perceive the model 
adoption process to 
be inclusive, 
informative, 
thorough, and 
adequate to 
generate buy-in. 


3: The teacher or principal describes a process that was 
generally inclusive, but with some minor shortcomings — 
perhaps greater efforts should have been made to include all 
stakeholders — but etherwise was informative. Interviewees 
express only very minor reservations about the process. 












2: The teacher or principal describes a somewhat imperfect 
process — including only some stakeholders, perhaps too 
short — but with some effort to acknowledge the importance of 
gaining buy-in 














1 : The teacher or principal described a process dominated by 
a closed circle of individuals, with little feedback from other 
stakehelders. 














0: The teacher or principal describes no stakeholder 
involvement. 










Comprehensiveness 


This construct 
reflects the degree 
to which 


4: The teacher or principal clearly describes a range ef CSR 
activities, including professional development, parent 
involvement, and instruction, as appropriate. 












stakeholders 
perceive the 
breadth of CSR, 


3: The teacher or principal describes at least one key 
component in detail, and demonstrates awareness of other 
components. 












or, in contrast, 
focus on a narrow 


2: The teacher er principal describes one or two components, 
but with thin detail. 












range of 
components. 


1 : The teacher or principal is aware of some terminelogy 
associated with the medel, but is unable to provide any 
additienal information about the model. 














0: The teacher or principal exhibits no awareness of the CSR 
model or associated activities. 
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The survey measures that were specifically associated with the adoption process were limited to a few 
items. First, the principal survey included items regarding whether the model had been adopted by a vote, 
through a consensus process, or through another means. If the principal indicated that the model had been 
adopted through a vote, the following survey question asked the percentage of faculty that had voted in 
favor of the model. Other model-related items sought determine whether the school adopted a model by 
choice or by mandate and whether the model was subsequently dropped. The teacher survey included 
similar items regarding the way in which the model was adopted. Finally, the district survey included 
questions to ascertain whether the district mandated implementation of one model or a preselected group 
of models and the type of technical assistance provided to schools during the model selection process. 

Thus, through the quantitative data, we were able to analyze the relationship between the following: 

♦ Whether or not teachers had the opportunity to vote, and the level of implementation 

♦ The percentage of teachers who voted in favor of the model and the level of implementation 

♦ The extent to which a critical threshold of teachers supporting the model was favorable for 
implementation. 

In the above analyses, the following variables were used: 

♦ The voting related variables (independent variable, whether voting took place) were derived from 
principals’ reports regarding voting related to adoption of CSR programs. 

♦ Threshold 80% = 80% or less of the faculty voted for adoption of the CSR. (145 schools had 
more than 80% voting for adoption.) 

♦ Threshold 90% = 90% or less of the faculty voted for adoption of the CSR. 

The threshold variables were created in this way because there were very few cases in which less than 
80% of the faculty voted for adoption of the CSR program (n = 24). In the threshold analyses, the 
following were used as covariates: 

♦ School-level variables: school size, free/reduced-price lunch, percentage non-English speakers, 
new principal, middle school, meet AYP, 3-5 and 5 or more years of implementation, adopted 
CSR 

♦ Teacher-level variables: usefulness of developer information, teacher community (social capital 
variable), teacher report of principal instruction, new teacher, English teacher 

In addition, we conducted selected analyses using hierarchical linear modeling. Two types of multivariate 
analyses were conducted: 

♦ The relationship between voting (yes/no) and the total level of implementation 

♦ The relationship between voting thresholds (80 and 90%) and the total level of implementation 

The relationship between voting variables and the total level of implementation were explored by using a 
two-level HEM model, including teacher and school levels. The models regarding the relationship 
between voting cut-off values and the total level of implementation were also conducted by using 
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multiple imputation to impute missing eovariate data. The substantive and statistieal results were the 
same, whether multiple imputation was used or not. 

Findings 

How Prevalent Are Specific Model Adoption Processes? 

How might adoption proeesses vary by eontext? As the coneeptual model suggests, it is indeed plausible 
that eontext — either of the sehool level itself, or the distriet in whieh a sehool is situated — would shape 
the proeess through whieh a sehool deeides to adopt a CSR model. For example, some distriets might 
openly eneourage sehools to vote, while others may eonstrain the proeess. Other eontextual variables, 
such as district or school size, may result in a greater likelihood that schools will engage in a voting 
process. 

District Characteristics 

Overall, our data suggest that voting for a model has become an institutionalized practice: in 8 of the 21 
NLECSR districts, all of the principals reported that their faculty engaged in a voting process to select the 
model. In seven other districts, over 80% of responding principals reported voting (see Figure 4.2). The 
efforts of CSR model developers to ensure that school stakeholders approve the adoption of a model 
appear to have become firmly entrenched in most districts. There was evidence of some minor district- 
level variation with regard to voting: in one district only 50% of the CSR schools voted to adopt their 
model, and in one third of the districts this figure was below 80%. However, no systematic variation was 
evident among these seven districts in which schools were less likely to engage in a voting process. 

Figure 4.2. Percentage of CSR schools in NLECSR districts that engaged in a voting process 












Few district-level demographic variables explain this (limited) variation in voting for a CSR model. As 
depicted in Figure 4.3, district size is somewhat related to the voting process. In districts with fewer than 
100 schools, 94% of the CSR schools reported having voted for the model that they subsequently 
implemented. In contrast, in districts with more than 300 schools, 76% of CSR schools reported voting. It 
is possible that in smaller districts, voting norms are more likely to pervade the district, or that 
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district-level efforts to support specific CSR activities are easy to monitor and enforce among fewer 
schools. None of the variation in voting was associated with poverty or student demographics. 

Figure 4.3. Teachers voting in favor of model adoption, by district size 
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Next, it appears that there is a weak relationship between CSR eoneentration (that is, the pereentage of 
sehools within the distriet that have an affiliation with a CSR model developer) and voting, although the 
effeet is not as pronouneed as for distriet size. In relation to voting, there is an apparent threshold at whieh 
sehools are more likely to vote: In distriets in which more than 35% of schools have a CSR model, 93.8% 
of such schools reported voting for their model. In districts in which less than 35% of schools have a 
model, the likelihood of voting is lower, at 79.6%. (See Figure 4.4.) 

Figure 4.4. Percentage of schools that vote for model adoption, by CSR concentration 
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Overall, a few district characteristics — size and CSR concentration — were weakly related to the 
likelihood that schools will vote for the adoption of a CSR model. However, the most relevant conclusion 
is that voting, as part of the process through which schools adopt a CSR model, has become a nearly de 
facto component of the life cycle of CSR. 

School Characteristics 

Turning next to the school level, we sought to determine whether there were any relationships between 
school contextual variables and the likelihood that a school’s model adoption process included a faculty 
vote. Again, we examined the relationship between the principal’s report of voting and demographics. 
Interestingly, almost no school-level contextual variables appeared to be systematically related to voting 
for a CSR model. 

The key findings of interest relate to school size. To investigate this, we used a simple logistic model in 
which the principal’s report of whether or not the staff had engaged in a voting process was the dependent 
variable, and the school size in the unit of 100 students was used as the predictor without other controls. 
These analyses suggest that larger schools were somewhat more likely to vote: according to this model, 
an increase of 1 00 students in school size increased the probability of voting by 6%. However, survey 
data also indicate that large schools were less likely than small schools to reach a high percentage in favor 
of adoption. The percentage of votes that favor adoption had negative correlation with school size: The 
correlation coefficient was -.128 and significant at the 0.05 level. 

One interpretation would suggest that there were challenges associated with communicating important 
model-related information within schools that have large faculties. Perhaps the process through which 
teachers were informed of model options was hindered within large schools, which resulted in lower 
levels of favorable votes during the decision-making process. Alternately, it was possible that it was 
simply harder to achieve a broad consensus within a large — and perhaps more diverse — faculty. 

However, the prevailing conclusion of these analyses is that voting for a model is widespread. 

Does the Adoption Process Have Implications for Implementation? 

Research on CSR suggests a probable association between the quality of the model adoption process and 
subsequent implementation. Having discussed the prevalence of the adoption process, we now explore the 
interaction between adoption and variables associated with implementation. 

First, many models (Success for All [SFA], Core Knowledge, Direct Instruction, America’s Choice) 
require that at least 80% of the school’s faculty vote for the adoption of their model before 
implementation, as evidence of teachers’ involvement in the decision-making process (Stringfield, Ross, 
& Smith, 1996). Indeed, all New American Schools (NAS) models required some threshold of teachers to 
vote for the model, although this threshold ranged from 60 to 80%. To what extent does a critical mass of 
teachers who support implementation actually signal true buy-in and result in improved implementation? 

Because our survey data included a measure of the percentage of teachers who voted in favor of the 
model, we were able to gauge the relationship between a specific threshold of teachers voting in favor of 
the model and level of implementation. In general, we found that principals reported very high levels of 
teachers voting in favor of the model — across the CSR schools in the NLECSR sample, the average 
percentage of teachers who voted in favor of the model was 88%. Perhaps because of these generally high 
levels of support, we found no relationship between level of implementation and a critical threshold of 
teachers in favor of the model, whether we used an 80% or even a 90% threshold. Indeed, at the 80% 
threshold, we found a significant negative relationship between level of implementation and the 
percentage voting in favor of the model. 
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Table 4.2. Percentage of schools in which the proportion of teachers voting in favor of model 
adoption was above or below specific thresholds 





80% threshold 


90% threshold 




Percentage of 
schools below 
threshold 


Percentage of 
schools above 
threshold 


Percentage of 
schools below 
threshold 


Percentage of 
schools above 
threshold 


Accelerated Schools Project (n=22) 


22.7% 


77.3% 


63.6% 


36.4% 


ATLAS Communities (n=6) 


16.7% 


83.3% 


50.0% 


50.0% 


Co-nect (n=10) 


30.0% 


70.0% 


60.0% 


40.0% 


ELOB (n=5) 


20.0% 


80.0% 


80.0% 


20.0% 


MRSh (n=4) 


50.0% 


50.0% 


75.0% 


25.0% 


SFA (n=89) 


34.8% 


65.2% 


68.5% 


31.5% 


TOTAL (n=136) 


31% 


69% 


67.4% 


32.6% 



There are some plausible reasons for which a critical threshold of 80% of teachers in favor of adoption 
was associated with lower levels of implementation. Most notably, some models that appeared to be 
easier to implement — that is, they had higher levels of implementation (see chapter 5) — also tended to 
receive lower levels of teachers who initially voted to adopt the model. SFA was the most notable of 
these, with only 65% of schools voting above the 80% threshold (see Table 4.2). Conversely, process- 
oriented models generally received higher levels of initial votes, but later appeared more difficult to 
implement. Overall, it appeared that there was little empirical support for a “super-majority” of faculty 
votes for the adoption of a model as an indicator of buy-in. 

It is also possible that the generally high proportions of teachers who voted in favor of the model masked 
the generally positive impact of the voting process. As Stringfield, Millsap, and Herman noted in 1997, 
“regardless of the abstract strengths of a reform, the fact that a principal and faculty had considered 
diverse options and voted to follow a particular path increased the probability of successful 
implementation.” Thus, it is possible that it is the act of voting that was important, rather than an arbitrary 
threshold. Perhaps if teachers had a voice, and a vote, this was enough to secure higher levels of 
implementation. 

Again, we found that there was no relationship between voting and implementation. That is, when 
comparing CSR schools that engaged in a voting process to those that did not vote, the effect of having 
involved faculty in a democratic voting process did not appear to yield any implementation benefits. 

However, we are cautious about discounting potentially positive effects of the model adoption process. It 
is possible that there are still some benefits of a high-quality adoption process that were simply not 
detected by our measures. Indeed, NLECSR survey instruments did not include measures of the 
legitimacy of the adoption process. Moreover, analyses did detect a significant relationship (at the 0.05 
and 0.01 levels) between voting and the teacher survey scale related to clear and shared goals (see Table 
4.3). If faculty members come together during the adoption phase, this is likely to foster (or perpetuate) a 
climate in which teachers are united behind a common purpose. And, because voting is indicative of an 
inclination to include teachers in a decision-making process, we would expect schools that voted to also 
be those with positive indicators of shared decision making. Indeed, the principal survey scale related to 
“school-level involvement in decision making” was significantly related to whether a school had voted for 
its CSR model (at the 0.05 level). 
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Table 4.3. Regression coefficients for survey scales and voting for CSR models 





Coefficient 


Standard error 


School-level involvement in decision making 


0.166* 


0.073 


Clear and shared goals 


0.149** 


0.027 


Usefulness of developer-provided information 


0.227** 


0.040 



*p value significant at 0.05 level. **p-value significant at 0.01 level. 



Finally, we found a significant relationship in one survey scale that was more closely related to 
implementation: teachers’ perceptions of the usefulness of information provided to them by model 
developers. This scale was derived from several items on the teacher survey that probe the efficacy of 
specific developer supports, such as a needs assessment, curricular materials, onsite assistance, or 
conferences. In cases in which they voted for a model, teachers had more positive perceptions of 
developer activities. Perhaps teacher involvement in the voting process predisposed them to a favorable 
impression of the model, or active engagement during the adoption process resulted in the selection of a 
model that best met the needs of a school. In either event, positive perceptions of the usefulness of 
developer-provided information bode auspiciously for implementation. 

To understand the model adoption process, such analyses are only a first step. Although these analyses 
reveal broad contextual relationships, the survey questions only addressed a single segment of the 
adoption process: whether a school voted for the CSR model. Survey data did not enable us to explore the 
degree to which teachers were informed of the model choices, who was involved in the adoption 
processes, and whether the processes were perceived to be legitimate and open. The qualitative data 
yielded more insight into the range of activities in which schools engaged as they select a model. 

When Schools Adopt CSR Models, What Is the Quality of That Process? 

To better understand the range of activities in which schools engaged when selecting a model, we 
carefully reviewed all interview transcripts from teachers, principals, and model coordinators and coded 
these data through the NLECSR Analytic Rubric. As depicted in Figure 4.5, there was indeed a large 
range within the “ratings” schools received for the aggregate quality of their adoption processes. 

While calibrating all scores, it became apparent that schools fell into specific clusters, based on the 
activities that characterized their adoption processes. That is, specific practices seemed prevalent in 
schools that fell within a limited range of adoption ratings, and these practices lent themselves to natural 
clusters. The descriptions of these processes are as follows: 

♦ Closed Adoption (ratings of 0.0-0.40). Schools with the lowest ratings were those in which the 
adoption process could be characterized as “closed.” In such schools, both the information- 
gathering and decision phases involved few staff; the former most often included a single 
individual or restricted set of staff, and the latter was either a forgone conclusion or teachers had 
no say whatsoever. Often (but not always), there is a lingering sense of negativity with regard to 
the adoption process in such schools. 

♦ Directed Adoption (rating of 0.41-0.60). In these schools, there was one organization or 
individual who strongly encouraged the school to adopt a model, most often the principal. Quite 
often a vote did occur, but it was conducted primarily because most models require a formal vote 
process and often lacked legitimacy. 
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Figure 4.5. Rubric ratings for the model adoption process 
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Closed Adoption 



Directed Adoption 



Guided Adoption 



Open Adoption 



♦ Guided Adoption (ratings of 0.61-0.80). In schools with guided adoption, the information- 
gathering phase was often quite open and inclusive, and the decision phase legitimate, but at least 
one key aspect was lacking. For example, a school may fall short with regard to the formats 
through which it provided information on the model choices. Instead of visiting schools and 
inviting model developers to visit, stakeholders may have only reviewed videos and text 
materials. In such schools, teachers had a generally positive recollection of the adoption process. 

♦ Open Adoption (ratings of 0.81 or higher). In these schools, there was clearly a proactive 
engagement on the part of many staff members with regard to model adoption. The information- 
gathering phase was extensive, often characterized by teachers visiting schools with models, 
making presentations to their colleagues, actively questioning model developers, and debating a 
range of model choices. The decision process was legitimate, and teachers were allowed the 
opportunity for dissent. In short, all aspects of the process were “open,” including participation, 
model consideration, and feedback. 

The following profiles further illustrate the ways in which these patterns took form at the school level. 

Profile of Open Adoption (Rating of 0.81 or Higher) 

Northway Elementary School^^ selected a CSR model in the context of a district initiative that 
encouraged schools with at-risk students to adopt models. Despite this potentially negative impetus for 
reform, staff at Northway has no negative perceptions of the adoption process. With regard to the 
information-gathering phase, faculty at Northway asserted that they had sufficient opportunities to learn 
about their selected model (SFA) in comparison to other models. School staff learned about the model by 
forming a designated committee that worked collaboratively with the school management team to 



’ All school names are pseudonyms. 
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consider different models and present information about the models to the staff They researched at least 
seven different models that included Comer, the Accelerated Schools Project, America’s Choice, and 
SFA. In addition, the district provided presentations on all the models. Finally, teachers reported receiving 
documentation on different models, reviewing these model data, and discussing them with colleagues. 



Formally, school staff at Northway selected SFA through a vote. School-level staff voted on all models, 
and SFA received more than 90% of the staff vote. Teachers perceived that they had authentic input in the 
decision process. Teachers in the focus group stated, “We had input at looking at the other models. . . . 

We went through them together, the administrators and teachers.” Years later, the staff was able to 
articulate their reasons for having adopted SFA. Teachers believed specific aspects of the model fit the 
needs of the students because it focused on reading for 90 minutes, required at least 20 minutes of student 
reading per day, and built connections between home and school. The reform facilitator concurred, noting 
that “Success for All really meshed with the teachers’ beliefs.” 

In summary, despite actions taken by the district to narrow model selection choices for Northway, the 
school-level decision making for the model was inclusive of school staff. School committees and the 
district worked to generate information about various models for school staff. School staff had a positive 
tone toward the model at adoption time particularly because of perceptions of fit with existing teacher 
beliefs as well as children’s needs. 

Profile of Closed Adoption (Rating of 0.0 to 0.40) 

At Baxter School, the adoption process was strikingly different. Most teachers at Baxter were excluded 
from the information-gathering phase of the process, which was restricted to a closed group of teachers on 
the restructuring team. The few teachers on the restructuring team were able to research different models, 
as one explains: “Our principal gave us a list of programs — at the time I was on the restructuring team — 
and we were allowed to go to demonstrations of other reform programs. ... I was impressed with 
America’s Choice; I thought it would have worked for us here.” But teachers who were not on the 
restructuring team were less engaged, as illustrated by the following quotes: 

“I was at the school when the model was adopted, but I was not involved in the selection 
process. ... I had no input.” 

“I think I had heard a little about American Choice [sic] ... or was it Success for All?” 

“We came back one school year [and] we were told this is what we’re going to do.” 

Most teachers were ambiguous about whether they were given the opportunity to vote for the model; the 
only interviewed teacher who recalled voting was the member of the restructuring team. However, she 
acknowledged that several teachers had lingering resentment with regard to the model, and these teachers 
later asserted, “I didn’t vote!” Other commentary from teachers suggests that they perceived that CSR 
was simply a mandatory process in which they had no say. Even if we accept some accounts that there 
was a vote, clearly it lacked validity to many of the faculty at Baxter. 

In sum, the adoption process lacked transparency — few teachers received any information about models 
under consideration, and fewer still perceived they had a voice in the decision. Not surprisingly, teachers 
at Baxter retain some bitterness about the model. 
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Case Study Data 

While the survey data reveal several promising assoeiations between voting and indieators of school 
climate, case study data enables us to more carefully probe the dynamics of the adoption process. To 
further explore the relationship between the adoption process and constructs related to implementation, 
we turn next to the qualitative data. The NLECSR Analytic Rubric captured several constructs closely 
related to the implementation of the model. Most notable among these is the comprehensiveness 
dimension. This is conceptualized as the degree to which stakeholders understand that the CSR model is 
intended to be comprehensive or multidimensional — that is, encompassing multiple aspects of school 
activities, including instruction, governance, professional development, and parent involvement, among 
others. When respondents were aware of several model components — and articulated this in the context 
of an interview — ^they received higher ratings for comprehensiveness. Clearly, knowledge and awareness 
of model components are important precursors to full implementation of the model. 

Figure 4.6 depicts the relationship between the quality of the adoption process and stakeholder 
understanding of model comprehensiveness, both as measured by the NLECSR Analytic Rubric. Results 
from these analyses may be used to cautiously explore emergent trends, with the primary objective of 
identifying relationships that merit further qualitative study. The addition of a simple regression line 
revealed a very weak relationship between the adoption process and a comprehensive understanding of 
the model. That is, as the quality of the adoption process increased, so did stakeholder awareness of key 
components of the model — but only slightly. 

Figure 4.6. Scatterplot of rubric ratings for adoption and comprehensiveness 




As our conceptual model suggests, if teachers learn about a reform approach during the adoption process, 
they are in a stronger position to build on this knowledge during the implementation process. However (and 
perhaps more interestingly), there are examples from our case study data in which a school received a 
relatively high rating for the quality of the adoption process, but received a low comprehensiveness rating — 
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or the converse, schools in which stakeholders exhibited a high level of understanding of the model, despite 
an adoption process that was lacking. To highlight these variations. Figure 4.7 depicts the relationship 
between adoption and comprehensiveness ratings in a slightly different manner. 

In Figure 4.7, the relationships among the ratings emerge more strikingly. Several schools had predictably 
close associations between adoption and comprehensiveness, with the ratings for both dimensions 
relatively low or relatively high. Interestingly, many of the schools with the lowest ratings for adoption 
had much higher ratings for comprehensiveness, and several schools with “open” adoption processes 
nonetheless exhibited lower levels of understanding of the model. To better understand the way in which 
school-level dynamics generate these disparate interactions, we explored these two sets of contrasting 
scores: schools with low adoption/high comprehensiveness ratings and schools with high adoption/low 
comprehensiveness ratings. 

Contrast A: Low Adoption/High Comprehensiveness 

Five schools had notable disparities in their adoption and comprehensiveness ratings — their 
comprehensiveness ratings were above 0.70, but the adoption rating was at least 20 points lower: 

♦ Centerville Elementary (adoption: 0.32; comprehensiveness 0.88) 

♦ Shoreland Elementary (adoption: 0.33; comprehensiveness: 0.70) 

♦ Iberville Elementary (adoption: 0.35; comprehensiveness: 0.73) 

♦ Chamberland Elementary (adoption: 0.58; comprehensiveness: 0.88) 

♦ Roseton Elementary (adoption: 0.56; comprehensiveness: 0.83) 

Figure 4.7. Comparison of rubric ratings for adoption and comprehensiveness 




booking across these schools, some common characteristics emerged, which inform understanding of the 
implementation process. Each school had either a strong, visionary principal, a relatively cohesive 
professional community, or both. In both Chamberland and Roseton Elementary Schools, teachers 
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described their principals in very positive terms, both emphasizing the principals’ determination. In 
Chamberland, for example, the teachers explained. 

Recently our principal was identified as a model principal. ... I think all of the things 
that we’ve been able to accomplish have been with the help or the leadership of [the 
principal], because she has vision and she does not sit back and wait for things to happen. 

If she hears about anything that’s coming down the pike, then she will dispatch people to 
learn as much as they can about it and bring it back here so that we are always on the 
cutting edge. 

Likewise, Roseton teachers explained, “There are certain things our principal is very passionate about. 
She’s always looking out for new ideas and the ones she thinks are going to come to fruition she pushes.” 
In these cases, the principals drove the adoption process because they thought the models would benefit 
the schools and successfully converted teachers to this vision, resulting in high comprehensiveness scores. 

In two of the other schools, the professional community facilitated faculty awareness and understanding 
of the models. Teachers at Iberville — one of the schools with a notably low adoption rating — described a 
cohesive community: “Lesson plans are presented not as an individual but as a group, and that keeps in 
mind that we are working together for the benefit of all the children and it’s not just one single pioneer, 
but all of us are pioneers together in this journey of education.” It appears that this sense of working 
together translated into a high level of model awareness. 

At Centerville Elementary, the qualitative data resulted in an overall adoption rating of 0.34 (one of the 
lowest) but stakeholders nonetheless exhibited a high level of understanding of the model (0.88). This 
was a school in which the principal clearly drove the adoption process, explaining, “Co-nect was really 
right in the line of what I was doing anyway. ... I wrote the grant.” Teachers were less engaged in the 
principal’s grant- writing process, recalling vaguely, “The model had already been selected when it was 
brought to the faculty. ... So, that’s how we found out about it.” However, this school eventually became 
a model for others that were considering adopting Co-nect, hosting visitors from other schools: 

They were visiting our school to see what we were doing and how we were doing it. And 
everybody was on target. It was just perfect. And the fact that we were able to connect 
the model with our daily activities made it all that more worthwhile. Because a lot of kids 
that probably would have been overlooked in some activities, they saw everybody else 
participating so they did too. . . . [The students] were really, really involved. You could 
really see the ownership of the children. 

In this school, however, it did not appear that the principal drove the success of the model, but rather the 
success was in the way teachers worked together. As one explained, “What I love about being here, the 
teachers work together. ... If we get lemons, we make lemonade!” Apparently, they did just that with 
regard to the CSR adoption process. This example suggests that an informative and inclusive adoption 
process is not a necessary precursor to effective implementation. 

In the final school in this category, Shoreland Elementary, the explanatory factor was less clear: The 
school was one that faced many challenges, including a transient community and a mildly acrimonious 
relationship between the principal and (at least some of) the faculty. The most positive element in the 
school appeared to be a shared sense of caring for the students. So here, too, it seemed probable that the 
professional community overcame the potential negative side effects of a closed adoption process. 
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Contrast B: High Adoption/Low Comprehensiveness 

There were three schools with adoption ratings higher than 0.64 and comprehensiveness ratings that were 
at least 22 points lower. Overall, there were fewer schools with extreme differences between high 
adoption process and low comprehensiveness ratings. The three schools in this category were: 

♦ Traceland Elementary (adoption: 0.64; comprehensiveness: 0.36) 

♦ Canton Elementary (adoption: 0.67; comprehensiveness: 0.45) 

♦ Greenway Elementary (adoption: 0.77; comprehensiveness: 0.42) 

Traceland Elementary offered an interesting example of this second set of contrasting scores. This school 
experienced a relatively inclusive adoption process, resulting in a rating of 0.64. However, teachers 
exhibited a relatively low level of understanding of the model, and the comprehensiveness score was only 
0.36. In this case, a strong principal served to ensure that the teachers were engaged in the adoption 
process and understood the model from the start. As teachers recalled. 

We had a very strong principal who was very interested in improving test scores. She was 
very interested in the research that was going on. She was a great networker and so she 
learned of [the model] and, you know, the philosophy behind it. . . . So she was kind of 
ahead of her time. She was always three steps ahead of what was going on, so that’s how 
she got us into it. . . . The staff really agreed to do the work. 

Another teacher said, “It had to be by faculty vote, so if we did not agree to it, they would not accept it. 

. . . The principal explained it quite well, and we discussed it quite a bit and then we agreed to it. . . . We 
saw a fdm; people came from the state and spoke to us.” However, after a few years of implementation, 
this principal left the school, to be replaced by an interim principal for 1 year. As a result, the components 
of the model faded quickly — and with some regret expressed by teachers who knew the former dynamic 
principal. Once this principal left, the faculty could not sustain the same levels of comprehensiveness, 
particularly as new staff joined their ra nks . As this case illustrates, the benefits generated by an effective 
adoption process and strong leader can dissipate quickly. However, had the principal not left, it is likely 
that this school would have retained a strong relationship between the quality of the adoption process and 
faculty awareness of the model. 

The data for Greenway Elementary offered several clues about barriers to implementation and the 
school’s low rating for comprehensiveness. Teachers expressed little enthusiasm for collaborative practice 
(said one, “We haven’t met that often collaboratively, but actually I prefer it that way”), contempt for 
district policies, and nothing more than lukewarm assessments of the principal. What was surprising, in 
fact, was that the adoption process was relatively inclusive and informative: teachers reported visiting 
other schools, studying models, deciding which model best suited their school, and voting for the model 
of their choice. However, this case suggests that even an adoption process designed to foster teacher buy- 
in may not be adequate to overcome school-level challenges to reform. 

To summarize, the qualitative data suggested that for some schools, there was a predictable relationship 
between the quality of the adoption process and the degree to which stakeholders understood the core 
components of the model — a key precursor to implementation. However, many schools in the case study 
sample deviated from this trend. For some schools a “closed” adoption process did not preclude eventual 
teacher buy-in; either a persuasive principal or a cohesive professional community could overcome this 
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hypothetical barrier. In contrast, an “open” adoption process did not appear adequate to overcome 
organizational challenges present within a school. 

Conclusion 

In the past, CSR researchers suggested that the quality of the adoption process was a critical precursor to 
gaining teacher buy-in, which in turn was an important indicator of the ultimate success of the model. As 
such, the “birth” of a CSR model had potentially important implications for success later in life. However, 
our data suggest a more nuanced view: adoption may be an important but not necessarily critical element 
in the implementation process. In some cases, strong school-level supports were of greater consequence 
than the quality of the adoption process. Overall, three themes emerged from the NLECSR analyses of the 
model adoption process. 

The first theme was that of the institutionalization of voting. Of the schools in the NLECSR sample that 
adopted a model, a clear majority did so through a voting process. As such, NLECSR data indicated that 
voting for a CSR model had become a fairly embedded practice. While there were a few schools that did 
not engage in a voting process, there seemed to be little systematic variation by school or district 
demographics. Only school size, district size, and CSR concentration explained this limited variation, but 
the effects of these variables were not particularly pronounced. 

The second theme was that of the limited relationship between voting and the level of implementation. 
Contrary to the assumptions of scholars and model developers alike, there appeared to be no benefits 
associated with a “super majority” of teachers voting in favor of the adoption of a given model. However, 
it is possible that schools that engaged in an adoption process that was informative, inclusive, and 
legitimate would have higher levels of implementation. Unfortunately, NLECSR survey data do not 
adequately address these constructs. 

The third theme was that of variation in quality. While some schools engaged in an adoption process that 
included many teachers, informed teachers of their options, and gave them an unfettered voice in decision 
making, other schools adopted models with little or no faculty consultation. Case study data indicated that 
the quality of the adoption process was loosely related to teachers’ perceptions of model 
comprehensiveness, a precursor to implementation. Survey data revealed significant relationships between 
voting and constructs associated with implementation, such as shared goals and perceptions of the quality of 
CSR developer supports. Despite these modest associations, the qualitative data were also interspersed with 
cases in which there were marked contrasts between the quality of the adoption process and teachers’ 
understanding of the model. 

Indeed, these contrasting cases suggest that the model selection process may be far less important than other 
school-level supports, most notably school leadership. In general terms, an “open” adoption process was more 
favorable for implementation, but an “open” adoption process itself appeared inadequate to overcome a 
challenging school culture. In addition, the benefits of an open adoption process may dissipate quickly if a 
charismatic school leader departs. Although NLECSR data did not enable us to conduct a full analysis of the 
relationship between the adoption process and leadership, they did suggest that the quality of the adoption 
process was a by-product of the quality of the school leadership. In cases in which the adoption process was 
closed, but comprehensiveness ratings were high, teachers frequently credited their principal with vision, 
determination, and a deep commitment to the school. In the (fewer) cases in which the adoption process was 
open, but comprehensiveness ratings were low, such a principal had recently departed. While past research 
has focused on the benefits of engaging teachers in the selection of the model, future research should explore 
the extent to which such benefits are primarily associated with the quality of school leadership. 
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Introduction 

The preceding chapter explored the initiation of reform in schools. This chapter continues the story of the 
CSR life cycle by examining what happens after formal adoption. That is, do schools that adopt models 
actually change practices in ways that are consistent with those advocated by model developers? Do they 
in fact implement the models? 

Measuring and accounting for implementation remains a key challenge for any program evaluation. The 
methods employed to measure implementation vary perhaps as much as the focus and content of CSR 
models themselves. Using both quantitative and qualitative analyses, we explore how CSR model 
implementation varies, identify what factors predict CSR model implementation, and illustrate how well 
school stakeholders understand CSR model implementation. The qualitative analysis of implementation 
captures the process and comprehensiveness of implementation from the points of view of principals, 
teachers, and staff in a subsample of study schools. Our approach for the quantitative analysis is quite 
different. We measure implementation as fidelity to the model design, or the extent to which schools 
conduct the specific school- and classroom-level practices recommended by a CSR model’s developer 
(Aladjem, 2003). 

The analyses included in this chapter focus on the following research questions guiding the NLECSR 
study: 

♦ RQ2 : How are model characteristics related to the success of model implementation and 
improvement in teaching and learning in specific types of settings and with specific types of 
students (RQ 2.1)? Specifically, to what extent does implementation of components of CSR vary 
overall and by type of setting (RQ 2.2)? 

♦ RQ3 : What supporting conditions and strategies are necessary to effectively implement and 
sustain CSR models in schools and school districts? Specifically, to what extent do states’ and 
districts’ CSR policies effectively support and maintain CSR (RQ 3.1) and to what extent does 
technical assistance provided by CSR model developers support CSR (RQ 3.2)? 



To address these general questions, we use both quantitative data and analyses and qualitative case study 
data to answer several more specific questions regarding CSR implementation. 
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Research Question 1: How Does the Implementation of CSR Practices Vary Across 
Schools and Over Time? 

To explore the variation in schools’ level of implementation, we first examine whether schools that are 
implementing CSR models engage in activities required by their CSR model to a greater extent than 
comparison schools that are not implementing the model.^'* We then investigate how implementation 
varies across CSR models to determine whether particular models tend to be implemented with greater 
fidelity than others. Next, since implementation is by nature an ongoing process, we study how schools’ 
fidelity to their CSR model design changes during the 3-year study period. We also assess the degree to 
which implementation varies according to schools’ phase of CSR implementation; that is, we check 
whether high levels of implementation are associated with schools that recently adopted a CSR model, 
schools that have been implementing a model for a few years, or schools that have been implementing a 
model for several years. 

Research Question 2: What Factors Influence School Stakeholders’ Implementation of 
CSR Practices? 

Our second research question probes the district-, school-, and teacher-level conditions that affect how 
CSR practices are implemented. First, for each dimension of implementation, we examine the amount of 
variance at the district, school, and teacher levels. Then, we test for variables that significantly predict 
schools’ level of implementation in 2002 and 2004. We also examine which factors are related to changes 
in schools’ implementation levels. Finally, we examine the contextual factors that school stakeholders 
perceive to influence their understanding of CSR models and ultimately the decisions that they make 
regarding implementation. 

Research Question 3: To What Extent Are CSR Models Perceived to Be Comprehensive 
In Nature, and Are They Implemented Comprehensively? 

Our last research question for this chapter considers whether, in keeping with the theory of CSR, schools 
are implementing model practices in a comprehensive manner. That is, we assess the degree to which 
schools are implementing multiple dimensions of CSR model practices, including areas ranging from 
instruction to school governance to professional development, rather than a select few. We also 
investigate the ways in which teachers and school leaders understand CSR models to be implemented 
comprehensively and on a schoolwide basis. We also search for differences between schools 
characterized by high or low comprehensiveness. 

Organization of the Chapter 

Next follows a discussion of the conceptual framework guiding this chapter, including a brief review of 
the literature on factors that may explain implementation. We then describe the methods and analytical 
approach used. The discussion of findings focuses on each of the research questions posed above. 

Theoretical/Conceptual Framework 

Characterizing Comprehensive Schooi Reform Impiementation 

Schools implementing CSR models are supposed to apply research-based approaches to every aspect of 
the schooling process. Building off prior federal statute, the 2001 No Child Left Behind Act defines CSR 

As explained below, the measure of implementation that we have developed allows us to measure 
“implementation” in our comparison schools; that is, it allows us to measure how “model-like” the practices are of 
the comparison schools. 
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models as reform programs possessing 1 1 key components, including a comprehensive design, research- 
based strategies, and high-quality professional development. These components are important because 
they are used as indicators to measure the programmatic adequacy of reform models, and they are 
required of every CSR model paid for through federal funds. 

Well-implemented CSR programs are assumed to profoundly change practices and activities taking place 
in schools, and the changes facilitated by CSR programs are intended to improve student outcomes. 
However, a recent meta-analysis of research studies on CSR’s effectiveness in boosting student 
achievement carried out by Borman and colleagues (2003) reveals a shortage of evidence that would 
permit developers to make even modest claims about the performance of their models. Moreover, it is 
clear that there is wide variation in models’ effectiveness in cases where evidence can be mustered. Both 
Borman’s work and the Herman et al. (1999) review suggest that among the great many CSR programs in 
operation, very few qualify as research-based. The reported lack of effectiveness and inconsistent findings 
could be due to varying level of CSR program implementation — any CSR effort is only as good as its 
implementation. 

Indeed, implementing a CSR program is hardly a straightforward process and the level of implementation 
tends to vary both within and across schools (Desimone, 2000). Moreover, many schools do not 
implement CSR programs comprehensively: only some of the core elements of the CSR program might 
be actively implemented at a particular time (Desimone, 2002; Berends, Kirby, Naftel & McKelvey, 
2001). For example, in RAND’s study of New American Schools (NAS), they found that only about half 
of schools began to implement the core components after 2 years of program adoption (Glennan, 1998). 
Bloom and his colleagues (2001) found that many Accelerated Schools began to implement components 
of curriculum and instruction in the 3rd or 4th year of implementation. Accordingly, it is likely that some 
schools will be more successful in their reform efforts than others. 



Factors Influencing CSR Implementation 

The success of implementing a CSR program depends on multiple factors (Desimone, 2002). CSR 
programs are implemented in a complex setting (schools) by a set of relevant school-level actors — 
administrators, teachers, CSR coordinators — who have varying capabilities to initiate and support a 
profound reform. The likelihood of successful reform — that is, full implementation of a CSR program and 
associated improvement in student achievement — involves actions of multiple actors on multiple levels: 
students, teachers, parents, principals, and district and state administrators. These factors include the 
context in which the reform occurs (school characteristics and district support for CSR), the roles of 
teachers and principals in the process of implementing a CSR model, and the developer supports for 
implementation. 

The Context in Which CSR Takes Place 



District support 

As discussed in chapter 3 of this report, districts can engage in a range of activities with regard to CSR. 
Some districts institute policies mandating the use of CSR or even particular models in their schools. 
Others may simply encourage schools to adopt a CSR model and provide various forms of technical 
assistance to support implementation. Still others may take a hands-off approach to CSR, leaving schools 
to locate their own sources of implementation support. 



RAND’s evaluations of schools implementing New American School designs determined that certain 
district contexts and activities were associated with higher levels of implementation of CSR models. 
Schools in districts that offered higher levels of support tended to exhibit higher levels of implementation. 
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Effective district-level support included direct activities such as providing resources and political backing 
for CSR as well as more indirect activities like promoting CSR- friendly school and district environments. 
Such environments featured a lack of crisis situations, a culture of cooperation and trust, school-level 
decision-making authority, and CSR-compatible accountability and assessment systems (Bodilly, 1998; 
Berends, 2001). 

Datnow and Stringfield (2000) recommend additional district-level supports for CSR, such as establishing 
a distinct set of common goals that are connected to key long-term improvement strategies; creating a 
coordinated effort for disseminating information about reform; facilitating a thoughtful, critical evaluation 
of what needs to change in a school and why; aligning policy systems to support reform; and being 
willing to change to adapt to the needs of the reform. 

School environment 

Not only does the level of CSR implementation appear to vary by district characteristics, but also by those 
of the school itself First, studies indicate that schools in high-income areas with smaller proportions of 
minority students have been more successful in implementing school reform programs (Berends, Bodilly, 
& Kirby, 2002; Yonezawa & Datnow, 1999). Moreover, student mobility has been shown to have a 
negative impact on implementation (Bloom et ah, 2001). 

Next, the district and state policy environment in which CSR schools are situated may affect they way in 
which models are implemented. Most notably, a school-level environment with conflicting or competing 
reforms or policies does not foster implementation with fidelity to the model, whereas a coordinated 
reform effort between different levels of actors (teachers, schools, school district, state) positively affects 
implementation of school reform (Hatch, 1998). Policies and reform programs with aligned goals and 
strategies facilitate implementation and avoid conflicts inherent in multiple demands. Schools that 
implement CSR programs often have high concentrations of students living in poverty, rendering the 
schools Title I-eligible, and hence eligible for many other state and federal funding streams. If states and 
districts do not carefully manage these funding streams, they may generate fragmented and conflicting 
environments (Berends et al., 2002, p. 135). According to Berends and colleagues (2002, p. 12), 
“[Teachers’] ability to cope with these demands and their commitment to changes are crucial to coherent 
and sustained implementation.” 

The Actors Who Implement CSR 

Several sets of individuals contribute to the success — or failure — of a given CSR model; among these are 
principals, teachers, and model developers themselves. Below, we briefly review the literature on the 
roles of each of these agents in the context of CSR implementation. 

Principals 

Principals who have deep understanding of the CSR program and who can adapt their leadership style to 
fit the needs of the reform can help the faculty to reach higher levels of CSR program implementation. In 
addition, principals who can secure additional resources, establish clear expectations, and are involved in 
everyday instructional decisions have positive impact on implementation of school reform (Berends, 

2000; Berends et al., 2002; Berends, Kirby, Naftel, & McKelvey, 2001). The importance of active and 
continuous principal leadership is demonstrated by the destructive effects of principal turnover: many 
reform programs do not survive the exit of a supportive principal. 

Teachers 

Although principal leadership has been linked to successful implementation of CSR programs, teachers 
bear most of the responsibility of successful implementation. Many CSR programs demand a high level 
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of professionalism, highlighting the importanee of informal teaeher networks as the sourees of knowledge 
and support necessary for successful implementation of a CSR program (Elmore, 1996; Purkey & Smith, 
1983; Bodilly, 1996; Stringfield & Datnow, 1998; Cooper et ah, 1998). These networks, or teacher 
communities, make collaboration and creation of clear goals and expectation regarding program 
implementation possible. Social networks and trust take time to build, and implementation of a 
demanding reform without a well-functioning teacher community may not be feasible. As a result, school 
reform efforts are likely to be negatively affected by teacher turnover. Further, teachers who have 
supportive professional networks and common planning time to tackle problems related to school reform 
programs are likely to implement programs with higher fidelity (Cooper, Slavin, & Madden, 1998; 
Stringfield & Datnow, 1998). 

CSR model providers 

Finally, CSR model developers, in addition to teachers and principals, are crucial actors in the successful 
implementation of a CSR program. Although developers support school stakeholders in a number of 
ways, such as providing information and monitoring implementation, the primary support activity is 
providing professional development. Ongoing professional development provided by CSR program 
developers has been identified as perhaps the most important aspect of support by building the necessary 
knowledge base and ensuring opportunities for teachers to collaborate (Desimone, 2000; Berends, 2000) 
as well as for the CSR provider to address potential teacher resistance to the reform (Haynes, 1988). 

Other researchers have also found links between successful implementation and professional 
development. For example, Berends and colleagues (2002, p. 15) found that levels of implementation 
were higher in schools that received design team support through whole-school training, the involvement 
of facilitators, and extensive professional development, compared to schools that did not receive such 
support. Professional development opportunities provided by the developer enable teachers to renew their 
knowledge and understanding of a model’s reform strategies. 

However, developers may also make mistakes in their efforts to stimulate change in low-performing 
schools. For example, Bodilly (1996) found that some developers had difficulty quickly securing staff 
needed to fully develop the model and to assist schools in implementation. In addition, developers 
sometimes moved too quickly, encouraging the implementation of a design before it was adequately 
developed, resulting in confusion during the first years (Bodilly 1996). 

Methods/Analytical Approach 
Data 

Quantitative Data Sources 

The quantitative data used in this chapter consisted of responses to NFECSR’s longitudinal teacher, 
principal, and district surveys.^^ Because our implementation measures are composed of teacher-reported 
information, we focus on data from Year 1 and Year 3 of our study (i.e., during the 2001-02 and 2003-04 
school years) when the teacher surveys were administered. For our control variables, we drew on data 
from all three surveys to measure factors concerning school and district contexts for CSR; we also 
incorporated some school information found in the Common Core of Data (CCD). To create our CSR 




See chapter 2 for more detailed information about our survey data collection. 
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implementation measures, we constructed 12 indices of teacher survey questions^® that probed various 
areas of school and classroom practice. We also asked CSR model developers to answer these survey 
questions, specifying the practices that would constitute full implementation of their model. The model 
developers’ responses were used to evaluate how closely teachers’ reported practices compared with full 
implementation of a particular CSR model. 

While our sample for the quantitative analyses featured both CSR and comparison schools, the inclusion 
of individual schools hinged upon our ability to compute their implementation scores. One major factor 
determining whether we could calculate these scores was the availability of model developer data. In all, 
seven model developers responded to our surveys; thus, the scope of our analysis was limited to schools 
associated with one of those seven CSR models. Additionally, several model developers omitted 
responses to individual survey items within our implementation indices, preventing us from measuring 
those indices for their particular CSR model. Our sample size also decreased as a result of our comparison 
school matching process. Since our analyses required linking comparison schools to compatible CSR 
schools within the same district, an imbalance in the response rates between districts’ CSR and 
comparison schools reduced the number of schools we could include. Also, because a small number of 
school districts have ardently adopted CSR models, we lost most comparison schools in these districts 
and consequently lost most of the CSR schools as well. 

Table 5. 1 shows how many CSR and comparison schools were used for each of the seven CSR models. It 
is important to note that some models — Success for All/Roots & Wings (SFA/RW), Co-nect, and 
Accelerated Schools Project — are represented more highly than others in our sample. However, these 
models tended to be particularly popular during our study period, and the unevenness of our sample to 
some degree reflects the variation in CSR models’ nationwide prevalence. 

Table 5.1. Number of CSR and comparison schools, by model and year 





Yearl 


Year 3 


CSR Model 


CSR Schools 


Comparison 

Schools 


CSR Schools 


Comparison 

Schoois 


Accelerated Schools Project 


29 


31 


25 


33 


ATLAS Communities 


12 


18 


13 


18 


Co-nect 


23 


36 


17 


31 


Success for All 


87 


84 


77 


70 


Turning Points 


6 


8 


5 


7 


Modern Red Schoolhouse 


8 


13 


7 


12 


Expeditionary Learning/Outward 
Bound 


5 


3 


5 


3 


Total 


170 


193 


149 


174 



’ We originally included several principal survey items in our implementation indices as well, but changes to our 
principal survey instruments made these items somewhat inconsistent between Year 1 and Year 3. Because 
teacher survey items covered most of the same information and teachers were likely more direct observers of 
school practices, we chose to base our implementation measures solely on teacher reports. 
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Qualitative Data Sources 

Our investigation of CSR implementation also relied on qualitative data, deseribed in ehapter 2, that we 
eolleeted through semi-struetured individual interviews and foeus groups with administrators and teaehers 
in 24 ease study schools. During the 2002-03 and 2003-04 school years, the research team conducted 57 
interviews of administrators (principals, assistant principals, and facilitators), 192 interviews of teachers, 
and 30 focus groups with teachers. In-depth interviewing with multiple school stakeholders provided an 
opportunity to discuss school processes and perceptions of CSR implementation (Spradley, 1979). 
Individual interviews of administrators lasted approximately 60 minutes, while teacher interviews and 
focus groups ranged from 15 to 30 minutes. During the interviews, the participants were asked questions 
about their implementation of CSR model components (i.e., school culture, organization and governance, 
curriculum and instruction, assessment, and professional development). Participants were also asked to 
discuss, among other topics, challenges and supports that they had received while they implemented the 
models. 

Analytical Models/Methods 

For our first research question, we conducted a descriptive analysis of our school implementation scores 
using two-way analysis of variance (ANOVA). The ANOVA models allowed us to test for significant 
differences in mean implementation levels between CSR and comparison schools among the various CSR 
model designs and across school districts. Furthermore, to determine whether implementation levels 
change over time, we tested for significant mean differences according to schools’ phase of 
implementation as well as between schools’ 2002 and 2004 implementation scores. 

While the ANOVA analyses indicated general trends in schools’ implementation levels, they did not 
control for other factors that could affect implementation. Thus, to investigate these general trends further 
and to locate implementation predictors in response to our second research question, we constructed 
multilevel regression models (three-level hierarchical linear models). These models — which included 
levels for teachers, schools, and districts — enabled us to determine which specific factors were 
significantly related to implementation, including school characteristics, developer assistance, or school 
districts’ policies in support of CSR. 

These multilevel models can be summarized as: 

Yy = + ^ + ^,jk + Yjk + ^k 



Where: 

Yij is the implementation index; 

the units of analyses are i teachers that are nested within j schools and k districts; 
a data matrix X contains values for the predictors; 
while R is a set of coefficients to be estimated; and 

errors — district-level error , school-level error , and teacher-level error — are normally 

distributed, with a mean of zero. 
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Due to the relatively small number of school districts, district-level support measures were added one by 
one. Moreover, we used two-level models (teacher and school) to predict change in implementation given 
the small amount of variance at the district level. 

The qualitative component of our second research question involved a thematic analysis across the CSR 
schools in our focus study to uncover conditions that may influence school stakeholder understanding and 
implementation of their CSR model. These analyses used interview and focus group data, employed 
cross-case data displays (Miles & Huberman, 1994), and were used to identify contextual factors that may 
explain school-level stakeholders’ variations in perceptions. The first step in these analyses was to code 
interview transcript data using the qualitative analysis software NUD*IST. Our predetermined coding 
structure matched constructs present in the NLECSR surveys, but we retained the flexibility to add new 
codes as they emerged in the open-ended data. We then compared the coded data on a school-level basis, 
drawing out themes that were present across multiple sites. 

Responding to our last research question, which investigates how comprehensively schools implement 
CSR, we again incorporate both qualitative and quantitative analyses. For our quantitative examination of 
comprehensiveness, we determined the number of CSR components (e.g., curriculum, inclusion, 
parent/community involvement) each school implemented at an above average level in 2002 and in 2004. 
Based on this number, we rated the school as having low (less than 4 components highly implemented), 
moderate (4-8 components highly implemented), or high (more than 8 components highly implemented) 
comprehensiveness in their implementation. We then conducted a chi-square analysis to search for 
differences in comprehensiveness over time and between CSR and comparison schools. To determine 
whether schools implemented instructional components of CSR comprehensively, we repeated this 
process using only 5 instruction-related implementation measures: inclusion, curriculum, student 
grouping, pedagogy, and time scheduling. 

Qualitative analyses for our third research question used the NLECSR Analytic Rubric (described more 
fully in chapter 4), focusing on its ratings for teachers’ awareness of the “comprehensiveness” and 
“schoolwide use” of their CSR model. The construct of comprehensiveness is conceptualized as the 
degree to which school-level stakeholders understood the model to be multidimensional — that is, not 
restricted to a single component such as instruction. Schoolwide use encompasses the degree to which 
stakeholders perceived the model to be implemented across classrooms, grades, and subjects. To 
accomplish this analysis, we delineated five distinct levels identified by a 0-4 rating scale and defined 
each carefully for both constructs (see Tables 5. 1 and 5.2). To complete a rubric for each school, a 
researcher read all principal, facilitator, and teacher interview and focus group transcripts, identified text 
that informed the constructs related to understanding of CSR, and rated each respondent’s comments for 
both constructs. 

We coded all of the data and created an appropriate construct score for each school. These scores were 
used to identify schools that had either high or low levels of CSR model understanding, and generated 
descriptive categories of schools in a continuum of CSR understanding. Later in this chapter, we profile 
two schools that represent each extreme of the continuum of CSR understanding — that is, exceptionally 
clear and detailed understanding of the components of the CSR model and an apparent lack of 
understanding of any of the core components of the model. The case studies profiles also illustrate the 
challenges and supporting conditions identified by stakeholders, and to which they attribute the success 
(or demise) of CSR implementation in their schools. 
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Measures 

Fidelity of Implementation 

The literature on CSR is replete with different strategies for measuring implementation. Typically, 
researchers have worked with model providers to determine a set of implementation benchmarks and 
indicators and then set about measuring them using school questionnaires (Bifulco, Duncombe, & Yinger, 
2005; Cook et ah, 1999; Supovitz & May, 2003), stakeholder interviews and classroom observations 
(Bodilly, Keltner, Purnell, Reichardt, & Schuyler, 1998; Bloom, Rock, Ham, Melton, & O’Brien, 2001; 
Datnow, Borman, Stringfield, Rachuba, & Castellano, 2003), or a combination of data collection methods 
(Smith et ah, 1997; Smith et ah, 1998). Other studies have relied on more indirect measures such as the 
number of years a school has been implementing a CSR model (Berends et ah, 2002) or schools’ 
self-reported degree of implementation (Kirby, Berends, Naftel, & Sloan, 2001). 

While assessing schools’ level of implementation tends to be relatively straightforward when evaluating a 
single program or intervention, measuring implementation across multiple models becomes a complex 
task. Berends, Kirby, Naftel, & McKelvey (2001) employed a clever approach in their study of NAS 
designs. They developed two survey-based measures of implementation: (1) a core implementation index 
that examined elements common to all NAS models such as parent/community involvement and 
standards-based assessments and (2) an intervention-specific set of indices that was geared to each 
individual intervention. 

To calculate the level of implementation for the diverse set of CSR models included in the NLECSR 
sample, we developed a somewhat different approach. We measure implementation as fidelity: the extent 
to which the CSR model of interest is delivered to the intended recipients in the intended way (Aladjem, 
2003). This approach (summarized in appendix A) is based on the idea that to measure the fidelity of 
implementation, we should measure what schools are doing and compare that with what CSR model 
developers consider to be “full” implementation of their reform. The challenge of measuring 
implementation is finding the difference between the positive, empirical reality of school life and the 
normative vision of CSR model developers. 

We operationalized this process by asking CSR model developers to fill out the same survey instruments 
as our principal and teacher respondents, as if they were a fully implementing school. Model developers’ 
responses to questions regarding school and classroom practices became what we refer to as CSR model 
keys, the sets of survey responses that define full implementation for each particular CSR model. 
Comparing our principal and teacher respondents’ answers against the key for their respective CSR 
model, we calculated the squared Euclidean distance between the ideal developer-specified 
implementation (as captured by the CSR model keys) and the actual implementation taking place in 
schools (as captured by the principals’ and teachers’ survey responses). To allow a more intuitive 
interpretation of the results, we then transformed this distance measure into a percentage of 
implementation (see appendix A for a full description of the methods used to calculate implementation 
indices). Thus, our fidelity of implementation measure can be understood to represent how fully a school 
is engaged in those practices that a school should be engaged in during full implementation of a specific 
CSR model. 

While the process just described provided implementation scores for schools formally implementing CSR 
models, we also needed to generate “implementation” measures for our nonimplementing comparison 
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schools^’ — that is, we sought to assess whether schools that had adopted CSR models were indeed 
engaging in different practices than schools that had not officially adopted a CSR model. Since measuring 
comparison schools’ implementation levels required comparing their practices against a specific CSR 
model key, we needed to determine which CSR model the comparison schools would be likely to 
implement if they were to choose to implement one. To assign comparison schools an appropriate CSR 
model, we decided to use a propensity score approach by which we calculated the propensity of being a 
CSR school for every school in our sample and then matched comparison schools with the CSR school in 
their district that had the most comparable propensity score (see appendix B for a full description of this 
method). As a result, we were able to compare levels of CSR model implementation between CSR 
schools and comparison schools with similar characteristics. 

The implementation indices used in our descriptive analyses are school-level measures; that is, we 
aggregated the implementation scores calculated for each teacher within a school to create a school-level 
measure of implementation. In the two-level HLM model featuring only teacher and school levels, we use 
teacher-level implementation indices. 

Our implementation indices include seven general categories and 12 specific implementation indices (see 
Table 5.2).^^ These indices align closely with the definition of “comprehensive design” included in the 
federal CSR grant program (20 USC 6516). The policy guidance for this program reads, “A 
comprehensive design for effective school functioning integrates instruction, assessment, classroom 
management, professional development, parental involvement, and school management. ... It aligns the 
school’s curriculum, technology, and professional development into a plan for schoolwide change” (U.S. 
Department of Education, 2002). 

Table 5.3 provides basic descriptive statistics for each of the implementation measures. In both Year 1 
and Year 3, the mean implementation levels for CSR and for comparison schools are strikingly similar 
across all indices. However, the mean levels of implementation do appear to vary considerably by the 
component indices. For example, the level of implementation is generally high on indices related to 
instruction, such as Curriculum, Time Scheduled for Instruction, Pedagogy, and Use of Assessments. 

The size of the standard deviations also varies by index. Larger standard deviations tend to coincide with 
indices that are less highly implemented such as use of technology, inclusion, and parent-community 
involvement. The size of these standard deviations implies less consistency in implementation across 
schools. 



Another valid method of testing whether CSR models change how schools operate would have been to compare 
the practices reported by CSR schools before and after they adopted their CSR model. However, since many 
schools adopted their CSR model before our surveys were administered, we lacked the data to perform such an 
analysis and thus relied upon comparison schools to provide this differential. 

Indices were weighted by how central and specific each dimension is to each CSR model (Forte Fast et al., 2001) 
Centrality and specificity ratings were determined by a review of published CSR model materials and information 
in 2000. 



AMKRICAN 

INSTITUTRS 

(OR Research 



Chapter 5: Implementing Change 



Table 5.2. implementation indices 



Total Implementation: A measure that combines all implementation indices into one comprehensive measure of 
implementation. 



Governance 

Shared Decision Making: The extent to which decision-making authority in a school is shared among faculty, staff, 
and administrators according to the CSR program developer’s ideal of shared decision making. 



Technology 

Use of Technology in Classrooms: The extent to which the ways teachers use technology in their classroom match 
the CSR program developer’s ideal. 



Parent-Community Involvement 

The extent to which a school’s actions regarding parent-school communication and community involvement match 
the CSR program developer’s ideal. 



Professional Development 

Emphasis of Professional Development (PD): The extent to which the PD received by teachers matches the CSR 
program developer’s ideal regarding the type (all grade-level teachers collectively vs. noncollectively) and emphasis 
of the PD. 

Engagement in Informal PD: The extent to which teachers’ engagement in informal PD matches the CSR program 
developer’s ideal. 



Assessment 

Influence of Assessments: The extent to which different types of assessments influence students’ grades, grouping 
decisions, adjusting curriculum, etc., according to the CSR program developer’s ideal. 

Use of Assessments: The extent to which the teacher is using classroom assessments according to the CSR 
developer’s ideal. 



Organization of Teaching/Classrooms 

Inclusion: The extent to which non-native English speakers and students with disabilities are mainstreamed in 
general education classes according to the CSR program developer’s ideal. 

Student Grouping: The extent to which students are taught in similar or mixed ability groups and how often these 
groups change according to the CSR program developer’s ideal. 

Time Scheduled for Teaching: The extent to which the frequency and length of instructional time matches the CSR 
program developer’s ideal. 



Instruction 

Curriculum: The extent to which teachers teach mathematics or English topics according to the CSR program 
developer’s ideal. 

Pedagogy: The extent to which teachers engage in different instructional activities according to the CSR program 
developer’s ideal. 
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Control Variables 

To determine faetors that predict schools’ fidelity of implementation, we added a range of control 
variables^® into our statistical analyses. Generally speaking, these variables fell into four basic categories: 
factors related to school context, teacher characteristics, CSR model information, and district context. 

♦ To control for school context, we included: 

■ Three variables related to schools’ student population: student enrollment {School size), the 
percentage of students eligible for free or reduced-price lunch {Percentage of students 
eligible for free or reduced-price lunch), and the percentage of English language learners 
{Percentage of ESL students) 

■ A scale variable that featured survey items about change in district policy and leadership, 
change in school leadership, school safety, and the physical condition of the school 
{Challenging Environment Index) 

■ Two variables related to principal leadership: a dummy variable recognizing new principals 
with 1 year or less experience in their current school {Principal recently joined school) and a 
scale variable that captures teachers’ attitudes about principals leadership skills {Principal 
Leadership) 

■ A dummy variable that identifies middle schools, i.e., schools with grades 6-8 {School has 
middle grades) 

■ A dummy variable indicating whether schools failed to make Adequate Yearly Progress 
{Missed AYP) 

♦ To control for teacher background, we included: 

■ A dummy variable identifying new teachers with 1 year or less in their current school {New 
teacher) 

■ A dummy variable distinguishing between English and mathematics teachers {English 
teacher) 

♦ To control for factors related to CSR treatments, we included: 

■ Two sets of CSR model-related variables: one that takes a value of 1 for specific CSR models 
{Accelerated Schools Project, ATLAS Communities, Co-nect, ELOB/TP, MRSh, SFA/RW) and 
one that assigns a particular CSR model design to respective CSR schools and matched 
comparison schools {ASP Key, ATLAS Key, Co-nect Key, ELOB/TP Key, MRSh Key, SFA/RW 
Key) 

■ Two variables indicating any changes in schools’ CSR status: a dummy variable showing 
whether CSR schools dropped or switched CSR models during the study period 
{Dropped/switched relationship with CSR model) and a dummy variable showing whether 
comparison schools adopted a CSR model {Adopted CSR model) 

■ A scale variable that captures teacher’s view about the usefulness of CSR model developer’s 
assistance {Usefulness of developer’s support scale) 



Details on these variables are presented in appendix B. 
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■ Two dummy variables to distinguish schools’ phase of implementation: one for schools that 
have been implementing CSR for 3-5 years {Middle implementation stage) and one for 
schools that have been implementing CSR for more than 5 years of CSR model (Late 
implementation stage) 

■ A dummy variable indicating whether schools had a CSR model coordinator or coach {CSR 
coach/coordinator) 

♦ To control for district context and CSR support, we included: 

■ Four scale variables measuring different types of CSR support provided by the district 
{District assistance with CSR model selection. District professional development for CSR, 
District technical assistance for CSR, District community outreach) 

■ A variable indicating the percentage of schools in the district that are implementing CSR 
models {CSR concentration) 



Results 

How Does the Implementation of CSR Practices Vary Across Schools And 
Over Time? 

CSR Schools Do Not Systematically Have Higher Levels of Implementation Than Their 
Matched Comparisons 

Given the theory underlying CSR that instituting a cohesive, developer-specified set of practices can 
improve school performance, one would expect to find schools affiliated with a CSR model engaging in 
practices prescribed by that model more often than comparable, unaffiliated schools. Flowever, our 
analysis revealed surprisingly few significant differences between CSR and comparison schools’ average 
implementation levels. As a whole, the comparison schools appeared to be implementing practices very 
similar to those of their matched CSR treatment schools (see Table 5.4). 

Several factors might explain the striking similarity between CSR and comparison schools’ reported 
activities. School districts may impose mandates or guidelines regarding school policies, such as those 
concerning curriculum, inclusion, professional development opportunities, and assessments. It is also 
possible that many practices advocated by model developers (such as the 90-minute reading block 
pioneered by SFA) have become mainstreamed, accepted as the best practices, and endorsed by districts 
to be implemented in schools. Comparison schools may choose to incorporate model-like behaviors they 
have observed in CSR schools or encountered through school networking activities. Moreover, staff who 
transfer from CSR-implementing schools to comparison schools may continue to use and even encourage 
others to use model-prescribed practices in their new environment. 

Implementation Varies by CSR Model Design 

Although few implementation differences seemed to exist between CSR schools and their comparisons, 
implementation did vary a great deal across the CSR “keys” representing developers’ specified CSR 
model design. In other words, schools’ implementation levels were clearly related to the particular 
“bundle” of activities that each CSR model provider endorsed — some models were generally 
implemented at high levels, while others were much lower. This finding implies that some CSR models 
advocate practices that are more difficult to implement, perhaps because they require a larger amount of 
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resources or because they deviate more significantly from mainstream practice.^® Furthermore, CSR 
model designs might affect implementation levels by emphasizing particular dimensions such as 
instruction or school governance over others. Schools may also choose to customize a CSR model or to 
implement only select components, thereby lowering their fidelity to the developer’s original design. 

Table 5.4. Results from descriptive analysis 
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Particularly innovative approaches may be more difficult to implement because they involve more substantial and 
deliberate changes in school behavior, whereas practices that have been mainstreamed may already be occurring 
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Figures 5.1 and 5.2 further illustrate our findings that there are remarkably few differences in 
implementation between CSR and comparison schools but numerous differences across CSR model 
designs. We can see both gross differences between models as well as differences regarding activities that 
may not be central to particular models. Differences also emerged among dimensions that may be 
controlled by school districts (such as policies regarding inclusion of special populations, parent- 
community involvement, and time scheduled for teaching). 



Figure 5.1. Implementation of CSR key, Year 1 of the study (2002) 
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CSR Models and Matched Comparison Schools 



Implementation Levels Change Over Time 

A comparison of 2002 and 2004 implementation scores revealed that schools’ total level of 
implementation increased significantly over time. Additionally, the majority of our 12 individual 
implementation indices exhibited some form of change. Six indices increased between 2002 and 2004 
(informal professional development, pedagogy, inclusion, student grouping, technology, and parent- 
community involvement), two decreased (shared decision-making and time scheduling), and five 
remained roughly the same (influence of assessments, use of assessments, curriculum, and emphasis of 
professional development). 
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Figure 5.2. Implementation of CSR key, Year 1 of the study (2002) 
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Because implementation levels in both CSR and comparison schools were generally higher in 2004, this 
increase in total implementation likely implies that CSR practices are becoming more mainstreamed. 
However, the changes in implementation we found between 2002 and 2004 may also point to CSR’s 
nature as an evolving process. Strategies involving considerable changes to existing practice may take 
time for schools to fully realize, producing eventual increases in implementation fidelity. Moreover, some 
CSR practices, perhaps those that are more resource intensive, may be difficult for schools to sustain and 
thus lead to decreases in CSR model fidelity. Fluctuations in implementation may also stem from 
contextual factors including school leadership or teacher turnover. Changes in district policy that require 
schools to alter their behavior may impact their fidelity of implementation. Conversely, rigid district 
policies that govern school behavior may prevent schools’ implementation levels from changing, which 
may explain why potentially district-controlled areas like curriculum, assessment, and professional 
development remained fairly static. 

Variation in implementation over time might also suggest that schools are concentrating on particular 
components at different stages of the CSR process. For example, schools might initially focus on 
implementing a model’s instructional strategies and then, once those practices are in place, incorporate 
other dimensions like parent and community involvement. Plus, schools may re-evaluate the 
appropriateness of their CSR activities over the course of their implementation and make adjustments 
accordingly. To test whether schools’ implementation levels varied at particular points during the reform 
process, we searched for differences according to schools’ “phase” or duration of implementation; that is, 
whether a school had been implementing a model for less than 3 years, 3-5 years, or more than 5 years. 
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Indeed, we found that in 2002, schools implementing for 5 or more years had higher total implementation 
levels than schools implementing for 3-5 years, which in turn had higher levels of implementation than 
schools implementing for less than 3 years. In 2004, the relationship was a bit less clear. The group of 
schools implementing for 5 or more years continued to have the highest level of implementation and were 
significantly higher than the schools in the less than 3 years category; however, there was no significant 
difference between the 5 or more years schools and the 3-5 years schools or between the 3-5 years 
schools and the less than 3 years schools. 

Implementation Appears to Vary Significantly Across Districts 

In both 2002 and 2004, implementation levels varied significantly by school district for all indices except 
student grouping. As suggested earlier, districts may have policies that heavily dictate school practices in 
certain areas such as curriculum, shared decision-making, or professional development. Given our method 
of calculating implementation fidelity, schools’ implementation levels for such areas would relate to how 
closely CSR model-prescribed activities coincided with district-prescribed activities. Additionally, given 
the significant relationships we found between implementation levels and CSR model designs, it is 
possible that this variation across districts can be attributed to the prevalence of particular CSR models 
within each district. The nature of our sample is such that for many of our districts, one or two of our 
seven CSR models clearly predominate. Thus, the apparent jurisdictional differences we have found may 
actually be picking up differences between CSR model keys. We explore districts’ potential impact on 
implementation more in-depth in the following section in which we seek to identify specific variables that 
affect implementation. 

What Factors Influence Schools’ Implementation of CSR Practices? 

Our previous analyses suggest that implementation varies by model design, by component, and over time. 
Next, we sought to determine whether implementation varies most by teacher, school, or district, and 
what predicts such variation. To examine the question of the level of variance in implementation, we 
estimated unconditional, three-level (teacher, school, district) regression models. Table 5.5 shows the 
percentage of variation for each implementation index in each of these levels. 

Teachers Account for Most of the Variation in Implementation 

As Table 5.5 illustrates, most of the variation in implementation levels happens at the teacher level. That 
is, even within a single school, teachers implement CSR models at different levels and in different ways. 
However, implementation scores regarding shared decision making, inclusion, and time scheduling each 
have significant proportions of variance at either the school or district levels. This seems reasonable as 
these are typically decisions made at the school or district levels, not by individual teachers. For example, 
the ways in which students with disabilities are included in regular classrooms is not solely a teacher’s 
decision, but generally involves administrators and other support staff As such, we would not expect 
most variance at the teacher level. Somewhat surprisingly, pedagogy also has a high proportion of 
variance at the school and district levels, indicating that teachers’ classroom instructional practices are 
fairly uniform within schools and districts. Coherent and aligned professional development activities or 
curricular strategies may explain this finding, but our data did not allow us to test this hypothesis. Also 
surprisingly, curriculum and assessment — often determined by the district — showed most variance at the 
teacher level. This suggests that despite district or school curriculum and assessment requirements, 
teachers are not necessarily following district or school policies. 
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Table 5.5. Location of variance in implementation 



Teacher School School District 



Outcome 


2002 


2004 


2002 


2004 


2002 


2004 


Total implementation 


66.4%*** 


69.7%*** 


20.3%*** 


23.1%*** 


13.3%*** 
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34.5*** 


25.1%*** 
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Note. * = p<0.1; “ = p<0.05; *** = p<0.01 



Next, we turn to variables that might predict this variation in implementation (see Tables 5.6.1 and 5.6.2). 

CSR Model Designs Influence Implementation. 

Confirming the results of our descriptive analysis, our regression models indicated that large differences 
in implementation exist across CSR models. These results were consistent over time (from 2002 to 2004). 
This variation can likely be attributed to differences in the model developers’ recommended practices 
since Accelerated Schools Project, ATLAS Communities, and Co-nect all differ significantly in design 
from SFA/RW. For example, with regard to pedagogy, each of the first three models emphasize less 
scripted, more complex forms of instruction, typically found in higher grade levels, while SFA promotes 
more easily implemented (due to their highly scripted nature) instructional practices. On the other hand, 
SFA is rather demanding with regard to parental involvement. 



Teachers and principals in our case study schools also saw the design of their CSR model as an important 
factor in explaining the level of implementation. Specifically, they described constructs that we refer to as 
programmatic fit and perceived model effectiveness. 
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Table 5.6.1. Predicted levels of implementation by dimension in 2002 
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= p<0.1; ** = p<0.05; *** = p<0.01; (+)(-)=CSR schools have higher or lower level of implementation respectively. 
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= p<0.1; ** = p<0.05; *** = p<0.01; (+)(-)=CSR schools have higher or lower level of implementation respectively. 





Table 5.7.1. Predicted levels of implementation by dimension in 2004 
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Programmatic fit 

Programmatic fit refers to the degree to which school and CSR model goals, programs, and philosophies 
of education are aligned. Teachers and principals emphasized the importance of this fit for 
implementation. “We thought we had good programs here,” said one principal. “We didn’t need to change 
our whole program.” This principal noted that the CSR model fit the instructional style of the teachers by 
emphasizing project-based learning. 

Programmatic fit can also include how well a CSR model aligns with other school initiatives such as 
district-mandated programs and state assessment requirements. School stakeholders often perceived 
conflicts between district-mandated programs and implementation of their CSR models. Teachers in one 
focus group explained, “The demands that are administered to us from the board of education in terms of 
our curriculum and how it should be laid out — those demands far exceed the demands for the [model] this 
year. How can we incorporate [the model] and also incorporate what is mandated by [the district]?” 
Teachers in another school had an answer. “The district says, ‘No, Tm sorry you can’t do that [use 
model-related curricula]. We have our own scope and sequence, and we have our own themes.’” 

The pressures teachers reported relating to state assessments follow the complaints commonly expressed 
about such tests and their impact on model implementation. “We have so much pressure on us to get these 
scores up that sometimes you get away from the model.” “When you have to deviate from your 
instructional program to bring in things that they need to know for this or that test, that takes 2 or 3 weeks 
out of the program . . . it’s just interruptive.” 

Perceived model effectiveness 

Given this pressure to raise performance on state assessments, it is not surprising that teachers and 
principals cited CSR models’ perceived impact on student achievement as an influence on their 
implementation. Where stakeholders perceived value and efficacy in their model, they reported stronger 
implementation. (Note that these perceptions do not necessarily match the empirical data reported in 
chapter 8 of this report.) In addition to student achievement outcomes, teachers and principals placed a 
high value on what they perceived to be a positive impact of their model on teacher professional 
community and instruction. Teachers stated that they felt supported by their colleagues during model 
implementation and that working and communicating together about the model, student work, and 
instructional practices enabled them to develop collaborative relationships and improve implementation. 
Teachers similarly reported that seeing positive changes in their own instructional practices as a result of 
using their CSR model strengthened implementation. 

The Support of Key Actors Matters for Implementation 

The results of our regression models reinforce teachers’ perceptions that collaboration among colleagues 
facilitated implementation. Our measure of teachers’ professional community was consistently and 
positively related to the level of implementation across multiple implementation indices, as were our 
measures of principals’ instructional leadership and CSR model provider technical assistance. Our 
analysis predicting the change in schools’ implementation levels suggests that principal leadership may be 
of particular importance. Of all the contextual variables considered, only one was significantly and 
consistently associated with the increases in implementation: a positive change in a principal’s 
instructional leadership. 

Together, these results illustrate the importance of key agents (principals, teachers, and CSR model 
providers) in the implementation process over other contextual factors such as schools’ size, poverty 
level, and limited English speaking population. Such findings imply that CSR models can be successfully 
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implemented in different environments if the relevant actors are sufficiently engaged in the CSR process. 
Interview data from our case study schools produced similar findings. School stakeholders underscored 
the roles of these actors in terms of principal leadership, new teacher induction activities, and CSR 
developer assistance, all of which they perceived to influence their schools’ implementation of CSR. 

Principal leadership 

Teachers and principals reported that the principal played an active role in ensuring that CSR was 
implemented faithfully. Two activities were found to be integral to the implementation process: making 
organizational changes to prepare schools for implementation and monitoring implementation. 

Preparation of schools included things like reorganizing the school day to accommodate the needs of 
particular models — for example, the 90 minute reading period of SFA — as well as ensuring that faculty 
have common planning time. Common planning time was often seen as “very helpful for new teachers 
coming in because they have that opportunity to share and grow.” Principals also were critical in 
encouraging and enabling teachers to participate in schoolwide decision making. Said one teacher, 
“Everyone is given their wings when they walk in the door and [are] allowed to fly. We vote on things, 
we talk about things, we make our decisions.” This type of shared decision making was often done 
through leadership teams, committees, and study groups. Another teacher commented, “We all make 
decisions together. . . . Even though there is a leadership team, those of us in mathematics and language 
arts, those that teach those particular subjects, we all get together, and we talk about what we need and 
what we’d like to have [happen].” 




Monitoring implementation was the other key form of principal leadership. It generally took the form of 
classroom visits and review of lesson plans. Classroom visits were the primary way of monitoring 
implementation. Some principals followed a systematic schedule, others visited every classroom, others 
used a less predictable and ad hoc approach of “dropping in” from time to time. One principal noted, “It 
doesn’t take you 50 minutes to observe a classroom to see if teaching is going on [sic].” This principal 
described stopping in classrooms throughout the day to observe instruction. Reviewing lesson plans 
allowed some principals to monitor instructional activities and the extent to which they reflected the CSR 
model. One facilitator commented, “[The principal] wants to know how [the teachers] tie the model into 
their lesson plans throughout the week.” A teacher explained, “We submit our lesson plans so our 
administration can see what we’re teaching.” 



New teacher induction activities 



School stakeholders stressed the importance of new teacher induction as critical to CSR implementation. 
New teacher induction activities were designed to encourage buy-in and to ensure that new teachers 
learned about the model. Typical of many teachers’ attitudes was that of the veteran teacher who reported, 
“Someone who has been here since the start of the process sees it differently than someone [who just] 
came in September.” Teachers and principals consistently thought that CSR models required extensive 
learning for new teachers, partly because models were new to new faculty and partly because of the 
difficulty of implementing some aspects of some models. As one principal reported, “It takes a good deal 
of time to really learn this . . . program, get entrenched in it, maneuver it, and make it exciting. First, you 
have to teach the new teachers how to teach the program before you can get into the deeper things with 
them.” 

CSR model provider technical assistance 



School stakeholders highlighted the types of model provider support activities they received at their 
school as integral to the implementation process. On the basis of their descriptions, two general types of 
activities emerged: ongoing support and on-site technical assistance. 
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School stakeholders repeatedly emphasized the value and importance of ongoing support from CSR 
model providers. This support included workshops, materials (such as videos and books), and national 
conferences that focused on model components and strategies for instruction. One principal gushed, 
“We’ve had numerous in-services, and they’ve been wonderful and we’ve gained a lot of insight into 
various facets of the curriculum.” While generally praised, principals also expressed frustration with 
specific developers. Another principal noted, “I think they can do more. I think that [for] the amount of 
money that we are paying them, we are not getting the services that we should have. I think there should 
be more technical assistance at the school, at least monthly.” 

This principal was not alone in the call for on-site technical assistance — indeed, many stakeholders 
believed that on-site assistance was critical to implementation. Common modes of on-site technical 
assistance include classroom observations and feedback and working with teachers individually or during 
grade level or study group meetings. Teachers valued the opportunities for direct answers to questions and 
support in addressing problems. Frustration, however, was also expressed about the lack of on-site 
support and the need for more frequent contact. 

Principals’ and teachers’ engagement with model providers and their collaboration with each other 
suggest that it is the cooperation among these three key groups that promotes faithful implementation. 

This coordination may hold particular significance for CSR given the wide breadth that distinguishes it 
from more isolated reform strategies. The concept of CSR calls for an array of actors to institute 
concerted changes to numerous areas of school operations. In our last set of analyses, we explore the 
degree to which schools reflect this notion of comprehensiveness. 

To What Extent Are CSR Models Perceived to Be Comprehensive in Nature, 
and Are They Implemented Comprehensively? 

CSR rests upon the belief that uncoordinated changes to various aspects of school operations cannot lead 
to sustained school improvement. Indeed, CSR is grounded in the proposition that school reform must be 
comprehensive; that is, that a unifying mission or vision must drive changes to all aspects of school 
operations. As such, in addition to studying whether schools implemented CSR with fidelity, we 
examined whether schools implement CSR comprehensively. 

To quantitatively evaluate schools’ comprehensiveness of implementation, we rated schools as having 
low, medium, or high comprehensiveness based on the number of CSR dimensions they were 
implementing at high levels, i.e., the number of implementation indices for which they scored above the 
average level for our sample. Schools with low comprehensiveness implemented less than four indices at 
high levels, schools with moderate comprehensiveness implemented four to eight indices at high levels, 
and schools with high comprehensiveness implemented more than eight indices at high levels. Figures 5.3 
and 5.4 illustrate these levels of implementation for CSR and non-CSR schools. They also show how the 
level of comprehensiveness changes over time. 
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Figure 5.3. Comprehensiveness of implementation, 2002 



Percent 




Most Schools Implement Only a Select Number of CSR Components. However, Their 
Implementation Becomes More Comprehensive Over Time. 

In 2002, 22.6% of schools fell into the low comprehensiveness range, 63.8% into the moderately 
comprehensive range, and just 13.6% appear to be highly comprehensive. In year 2004, the overall level 
of comprehensiveness improved such that only 17.6% were of low comprehensiveness. The middle group 
also declined to 59% and almost one quarter of the schools (23.8%) had moved to being highly 
comprehensive. 

CSR Schools Tend to Implement More Components Than Comparison Schools. 

While most schools fell into our low or medium comprehensiveness categories, overall, schools 
implementing a CSR model were more likely to have a high comprehensiveness level than were 
comparison schools, both in 2002 and 2004. For the 2004 data, the differences between CSR and 
non-CSR schools were statistically significant at the 0.1 level. The generally low level of 
comprehensiveness could be attributed to selective implementation: schools may decide to implement 
only particular components of the CSR program, or schools may decide to implement different 
components sequentially, one after another. 
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Figure 5.4. Comprehensiveness of implementation, 2004 
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CSR Schools Implement Instructional Strategies More Comprehensively Than 
Comparison Schools. 

To investigate comprehensiveness further, we examined the level of comprehensiveness of classroom 
instruction. The rationale for this is based upon the notion that although all components are integral to 
CSR, instructional activities are presumably most likely to have direct effects on student achievement and 
consequently schools would reasonably be more likely to focus on those activities and implement them 
more highly and comprehensively. 

The composite measure of comprehensiveness of classroom instruction was created from the 
implementation indices for inclusion, curriculum, student grouping, pedagogy, and time scheduling. 
Values for this measure range from 0 to 5. About 2% of the schools in both years (2002 and 2004) had 
values of zero. About 34% implemented four or more components at or above average levels in both 2002 
and 2004. There were statistically significant differences between CSR and non-CSR schools in 2004, at 
the 0.1 level. CSR schools were implementing classroom instruction-related practices more highly than 
non-CSR schools. Figures 5.5 and 5.6 illustrate these findings. 
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Figure 5.5. Comprehensiveness of implementation of classroom practices, 2002 
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Figure 5.6. Comprehensiveness of implementation of classroom practices, 2004 



Percent 

40 



30- 



20 - 



10 - 



1.92% 

714Mb 



9.89% 
44 44% 

S5.Sa% 



32.42% 



22,53% 45.76% 



41.46% 



54.24% 



Se.54 



21.15% 



51.95% 



48.05% 



12.09% 



58.09% 



4a91% 



0 1 2 3 4 5 

Comprehensiveness of Implementation of Classroom-Related Practices 
CSR School H Comparison School 




AMERICAN 
INSTITUTES 
FOR Research * 





Chapter 5: Implementing Change 



We also drew upon qualitative case study data to examine “comprehensiveness” of implementation, using 
ratings generated through the NLECSR Analytic Rubric. Figure 5.7 depicts the comprehensiveness 
ratings for each case study school. Figure 5.8 depicts the schoolwide use ratings for each case study 
school. 

Figure 5.7. Comprehensiveness ratings for case study schools 
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Most School Stakeholders Demonstrate Knowledge of CSR’s Comprehensive and 
Schoolwide Nature. 

We see in Figure 5.7 that most of the ease study sehools rate over 0.7 on the comprehensiveness scale and 
well over 0.7 on the schoolwide use scale. There is somewhat more variation in the ratings for the 
comprehensiveness scores than for schoolwide use: while the comprehensiveness scores range from 0.27 
to 0.92, the schoolwide use scores range from 0.38 to 1. Drawing from these aggregate ratings, as well as 
specific aspects of the data from each of these schools, we categorized schools’ combined scores into four 
groups to describe respondents’ perceptions of their CSR models: acquainted, informed, knowledgeable, 
and fully articulate. 

♦ Acquainted (0.25-0.40) — School-level stakeholders hold awareness of terminology associated 
with the model and limited understanding that CSR extends beyond personal activities. 

♦ Informed (0.41-0.60) — School-level stakeholders hold awareness of CSR as one component in 
depth, and implementation is perceived across a few classrooms or a subset of teachers. 

♦ Knowledgeable (0.61-0.80) — School-level stakeholders hold detailed awareness of at least two 
components in depth and that CSR at least spans more than one grade, but they fall short of 
expressing CSR as a schoolwide initiative. 

♦ Fully Articulate (0.81-1.0) — School-level stakeholders hold detailed awareness of more than two 
components of the CSR model and that CSR is implemented across all grades and classrooms. 

Ninety percent of the case study schools fell into the informed, knowledgeable, and fully articulate 
categories for comprehensiveness. Only the remaining 10% were in the acquainted category. Eighty-six 
percent of the case study schools fell into the knowledgeable and frilly articulate categories for 
schoolwide use. The remaining were split between the informed and acquainted categories. Thus, in the 
majority of our case study schools, stakeholders exhibited moderate to high levels of understanding of the 
CSR models they had adopted. While understanding is not tantamount to implementation, it is a necessary 
precursor to successful implementation. 

To investigate school conditions associated with varying levels of CSR consistency and 
comprehensiveness, we identified two schools that highlighted the extremes. Chamberland fell into the 
fully articulated category for comprehensiveness and consistency (0.88 and 1.0, respectively). Ivyton fell 
into the acquainted category for comprehensiveness and consistency (0.27 and 0.38, respectively). These 
two cases are described in-depth in the text boxes below. 
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A Case of Fully Articulated Comprehensiveness and School-wide Use: Chamberland 

Chamberland Elementary is located in a high-poverty area of a large urban district, yet benefits from unusually high 
loyalty from parents, faculty, and administrators. Because public housing near the school was demolished, families 
moved out of the attendance zone, yet sought to keep their children at Chamberland. Teachers, too, expressed 
commitment to the school, with many teachers remaining at Chamberland until retirement or promotion. The principal 
commented, “Once they get here, they don't want to leave.” The principal herself had spent her entire career at 
Chamberland, beginning as a substitute teacher in the 1960s and becoming principal 30 years later. 

When NLECSR researchers visited Chamberland, the school had been implementing their CSR model for eight years, 
although funding for the model had ceased in 1996. Nonetheless, school-level stakeholders still expressed 
understanding of the model's core components. In particular, they described the use of committees as central to their 
decision-making efforts. As teachers explained, “all decisions are made [by committees]— those areas are academic 
performance, community involvement, school spirit, and discipline. Any problems that come about in that way, those 
problems are sent to that committee. Then the committee tries to work them out, and they send a report in turn to the 
principal with the suggestions that we have made. Everyone is on at least one committee. We meet once a month.” 

Initial training from the model developer established an important foundation for CSR implementation at Chamberland. 

In addition to off-site training, model trainers came to the school to instruct the teachers. According to the principal, 

“Then we had a coach who was also trained and was in the building every week, assisting and training the teachers.” 
This training over the first three years consisted of 2-day in-service sessions on every phase of the program. In addition 
to that ongoing professional development, the program model required one of the school's teachers to be an onsite 
coach for model implementation, thus assuring onsite technical assistance. 

A supportive professional community of teachers and strong principal leadership were also associated with the high 
level of awareness of the model and sustained CSR practices. Although rare, when teacher turnover did occur, school 
staff at Chamberland worked to ensure that new teachers were trained properly. Experienced teachers said that they 
tried to get the newer teachers to attend workshops so “they will learn what the rest of us already know.” The 
combination of low teacher turnover and support for new teachers to learn about the model encouraged a deeper 
understanding of model components. The principal also monitored implementation of the model by visiting classrooms 
and encouraging the use of model specific instructional practices. In addition, teachers explained that, “The principal has 
an open door. Anything that you want to bring to the table is up for discussion.” 

At Chamberland, teachers believed the model had helped their schools. Teachers thought students benefited more from 
lessons because the model encouraged them to make connections between their everyday experiences and the 
instructional content. One teacher commented that her students enjoyed mathematics more since the model began. 
When asked if the model helped to achieve academic goals, a teacher reported, “Yes, because we are still using it now!” 
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A Case of Acquainted Understanding: ivyton 

Ivyton Elementary is located in a high-poverty area which the principal described as an “urban blight area” with few 
community resources. Like many urban schools, Ivyton experienced high levels of teacher turnover. In 2004, for 
example, the principal reported that 85% of her staff was new and that “My staff, currently, is 80% under the age of 27, 
so they are just like brand new.” 

When NLECSR researchers visited the school, Ivyton had been implementing ATLAS Communities for 6 years— 
although the current implementation had tapered to nearly undetectable levels. When they first started model 
implementation, it the principal described implementation of the model at Ivyton as intense: “It was ATLAS Communities, 
ATLAS Communities, ATLAS Communities.” Six years later, however, only the principal of the school had much 
knowledge of the model. She commented that many teachers might not recognize the name of the model but would 
recognize the parts of the program— the few practices that persisted. Indeed, one of the few teachers familiar with the 
model noted that, “There is a trickle here and a trickle there, but it's not the model [we used to have].” 

The principal and those few teachers with awareness of the model attributed the model's demise to the high rate of 
teacher turnover. The principal conceded that most teachers would not recognize the model or articulate the 
components, although “15% could still say, ‘from A to Z, that this is the model.”' Another teacher observed, “People 
haven't been educated as to what [ATLAS Communities] is. And our school has a very high turnover rate for teachers.” 
One teacher in the focus group who had heard of the model stated, “Now, [ATLAS Communities] was brought up this 
year. In the beginning of the year, they brought up the model. But I don't remember anything happening from there.” 

The school stakeholders at Ivyton did not describe any developer activities at the school. Professional development 
activities from the developer may have occurred during the model implementation phase, but the teachers were not 
receiving any model-related support at the time of the site visits in 2003 and 2004. Moreover, the principal did not 
describe any method for monitoring or supporting the implementation of the model components still in place at Ivyton. 

Ivyton elementary is an example of a school in which there was limited understanding of the model and school 
stakeholders were vaguely familiar with model-related terminology. However, Ivyton faced a number of school-level 
challenges that influenced model understanding and implementation, the most dramatic of which was teacher turnover. 
This turnover rate is problematic in relation to reform because few of the teachers initially trained in the model remained, 
and the school no longer received professional development opportunities from the model developer. Furthermore, no 
structure appeared to be in place to inform the new teachers of model activities. 




Conclusion 

Changing practices in traditionally disadvantaged schools is hard work, which requires time, resources, 
and ongoing support. However, the analyses presented in this chapter indicate that it is indeed possible to 
effect change in low-performing schools in urban environments. 



Most importantly, NLECSR data indicate that individual CSR models do stimulate different practices in 
schools. Thus, the specific set of activities advocated by model developers — that is, the essence of the 
model design — shape school-level activities in unique ways. Some models are implemented at higher 
levels than others, and some components are more likely to be fully implemented, but CSR does seem to 
be an effective mechanism for stimulating change. 

Not surprisingly, implementation increases over time, with the highest levels of implementation in 
schools that have had their model in place for 3-5 years. Thus, while CSR can indeed stimulate change, it 
will be most pronounced if given adequate time to mature. Moreover, models will become more 
comprehensive over time — that is, schools appear to phase in model components over successive years. 

While CSR has clearly changed practices in specific schools that formally adopt models, the set of 
assumptions that frame CSR appears to have permeated the educational landscape. That is, practices that 
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are endorsed by CSR model developers have become widespread, and were detected even in comparison 
schools in the NLECSR sample. 

Changing practices is an important step in school improvement. But to what extent do these changes in 
practice stimulate improved student achievement, and to what extent can these strategies be sustained? 
The following chapters address these important questions. 




Chapter 6: Building Social Capital 

Kazuaki Uekawa 
Yu Zhang 
Daniel K. Aladjem 



Introduction 

This chapter explores the relationship between the implementation of CSR models, social capital, and 
student academic achievement. Most credit Coleman (1988, 1990b) with conceptualizing social capital 
and applying it to educational settings. Coleman defined social capital in the context of education, and 
this chapter attempts to examine it in the context of CSR. 

We review the broad and multifaceted literature on social capital and present an argument for the 
relevance of establishing a linkage between the social capital of teachers, CSR implementation, and 
academic outcomes. This chapter briefiy reviews studies that address the social capital of teachers 
implementing CSR models, and argues why teacher social capital may be expected to improve student 
achievement. 




The analysis section of the chapter presents the results of two separate analyses. The first analysis 
presents the results of measuring social capital among teachers as an outcome of CSR implementation. 
The second analysis presents the results of using social capital as a mediating variable between 
implementation and academic achievement. The chapter concludes with policy recommendations. 

Theoretical Framework 

The concept of social capital is thought of as a resource grounded in social relations (Burt 1992; Coleman 
1988, 1990b; Granovetter 1973). Having social capital means understanding norms that people adhere to, 
and understanding the trust that links people in a variety of social settings. Having social capital may 
mean many things, including the presence of social ties (weak or strong) and the nature of such ties 
(cross-cutting or dense). In this chapter, we present analyses on social capital in the framework of CSR. 
Within the context of CSR, we view social capital as composed of two distinct parts: professional 
learning community and collective commitment. 



This chapter focuses on these two dimensions of social capital — ^professional learning community and 
collective commitment — and we hypothesize that they are important to the improvement of instruction 
and student achievement. Professional learning community as a measure of social capital within schools 
includes the existence of collegial ties that are formed with the goal of improving instruction. Professional 
learning community should imply that teachers will meet, engage in formal and informal discussion 
groups and settings, and even feel a sense of cooperation. These networks allow for the creation of norms 
and expectations, and for the exchange of information about teaching. They also help to create conditions 
for social influence such that teachers using effective methods may be able to influence others to do the 
same. Collective commitment implies the presence of an agreement among teachers to engage in certain 
reform activities. Collective commitment as a measure of social capital posits the existence of stronger 
norms and expectations about student performance as a result of collective and consistent perceptions of 
the school’s mission. In sum, the professional learning community perspective explores the ties formed 
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among teachers regardless of their opinions, and collective commitment examines the views and opinions 
formed and held by teachers regardless of the ties. According to social capital theory, both professional 
learning community and collective commitment among teachers are expected to affect student 
achievement. 

The concept of social capital in school settings is not new. Most prior work has been descriptive in nature, 
and few studies have used comparison groups. Newmann, King, and Youngs (2000), for instance, argued 
that professional development is most successful when the school has a professional community that 
allows teachers to collaborate to improve their teaching. This assertion, however, is mainly supported by 
qualitative data and descriptive statistical analyses, with no comparison groups. Louis, Marks, & Kruse 
(1996) were able to demonstrate how a school’s professional community is best fostered under specific 
conditions of the schools. Examples of these conditions included staffing patterns, scheduled planning 
time, and teachers’ sense of empowerment. These studies revealed that the existence of a community can 
increase a teacher’s sense of responsibility for student learning. 

However, true community is often difficult to achieve within schools. Indeed, schools have generally 
been characterized as loosely coupled organizations that lack coordination among the parts in the system 
(Weick, 1976). This means that it is possible that leadership may be autonomous, and what the principal 
or district prescribes may not necessarily impact what happens inside the classroom. Studies of the 
occupational ethos of teaching further suggest that teachers are likely to favor working with students 
rather than collaborating with their colleagues. Lortie (1975), for instance, states that teaching is more 
about collecting psychic rewards from interacting with students, rather than working with colleagues 
(pp. 192-1 96). 

Additional studies of sociology in education (Bryk & Schneider, 2002; Miller & Rowan, 2006) have 
attempted to examine the links between students’ academic achievement and social capital, but these 
studies do not discuss the role of school reform or specific school reform strategies. Bidwell and 
Yasumoto (1999), for instance, used social network analysis to explore the links between student 
achievement and social capital. Their findings challenged the notion that teachers normally act alone, and 
found, instead, that teacher networks influence how teachers engage in teaching. Yasumoto, Uekawa, and 
Bidwell (2001) extended the analysis of Bidwell and Yasumoto (1999), and their findings revealed that 
the density of communication among colleagues was related to the effectiveness of certain pedagogies 
and strategies, which in turn could affect academic growth. Thus, in essence, the results revealed that the 
social network of a school may be related to student academic achievement. In this chapter we explore 
these phenomena further in the context of CSR. 

One question that remains unanswered, however, is why some schools experienced greater levels of social 
capital than others. Studies employing quasi-experimental designs that explicitly compare treatment and 
control schools may provide insight into the ways in which school reforms affect social capital. Such 
studies could also help us to understand how social capital may play a role in school reform, especially if 
these types of reforms affect student academic achievement. The next section revisits and reanalyzes 
social capital theory and introduces the hypotheses to be analyzed in this chapter. 

Hypotheses 

Analyses in this chapter address three hypotheses that explore the relationship between social capital, 
implementation of CSR, and student academic achievement. The first hypothesis is predicated upon the 
relationships between professional learning community and collective commitment. This hypothesis 
posits that social capital will affect the academic achievement of students enrolled in schools that adopt 
and implement externally developed CSR models. The second hypothesis measures the expected li nk 



.XMKRICAN 

INSTITUTF.S 

Research 



Chapter 6: Building Social Capital 



between CSR and soeial capital, by examining schools as loosely coupled organizations. The third 
hypothesis predicts that social capital may serve as a mediating factor for academic achievement and 
implementation. 

Research Questions 

This chapter addresses three research questions: 

RQl : To what extent does the implementation of externally developed CSR models affect the level of 
social capital in schools? 

RQ2 : To what extent does the implementation of specific CSR models affect the level of social capital in 
schools? This question seeks to test whether any CSR effect detected in answering RQ 1 can be traced to 
specific CSR models. 

RQ3 : To what extent is social capital associated with student achievement? 

The next section will address the data sources and the analytic approaches used to examine the data. 

Methods/Analytical Approach 
Data Sources 

Although the full NLECSR school sample included 649 schools, this chapter draws on data from only 359 
schools. The reduction in sample size resulted from the calculation of implementation scores (described in 
chapter 5). A comparison of the analytic sample of 359 schools and the larger sample showed not 
significant differences between the two social capital measures and mathematics and reading assessment 
scores. The teacher implementation measures are listed below (see chapter 5 for a full description): 

♦ Organization/govemance 

♦ Professional development/Technical assistance 

♦ Curriculum 

♦ Instruction 

♦ Inclusion 

♦ Technology 

♦ Time scheduling 

♦ Student grouping 

♦ Assessment 

♦ Data-based decision making 

♦ Parental involvement 




Analytical Models 

The first research question probes the relationship between the implementation of externally developed 
CSR models and the level of social capital in schools. To answer this question, we analyzed the two 
measures of social capital (professional learning community and collective commitment) separately. We 
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used a series of dummy variables to represent CSR models and eontrolled for important variables, 
ineluding soeial eapital measures from the prior year. For this researeh question, and for the remaining 
researeh questions, we used multilevel regression models. 

The seeond researeh question asked the extent to which any observed CSR effect on social capital can be 
traced to specific CSR models. To test this hypothesis, we modeled the interaction between the level of 
implementation and comprehensiveness of implementation across schools. First, we modeled the 
interaction to explore whether these interactions would have an effect on the overall measure of social 
capital. Second, we did this to investigate whether including both the level of implementation and the 
comprehensiveness of implementation would reduce the effects of social capital. The underlying research 
hypothesis was that that some CSR models may actually promote social capital, and therefore proper 
implementation alone could account for these changes. 

The third research question asked the extent to which social capital is associated with student 
achievement. The hypothesis associated with this question predicted that increases in social capital would 
lead to a change in student achievement scores from the years 200 1 to 2002, both in mathematics and in 
reading, although the data would be analyzed separately. This question was explored analytically by using 
social capital, a series of dummy variables representing CSR models, as well as other covariates and 
achievement scores also used as dummy variables for the baseline academic achievement scores. 

Schools were the primary unit of analysis for this chapter. The data were collected on different levels 
(principal and teacher levels) and aggregated to the school level to generate school-level scores for 
various measures. Finally, academic achievement scores in mathematics and reading were obtained from 
participating districts, and these data were also aggregated at the school level. 

Data were aggregated and then prepared for analyses by combining the outcome measures collected in 
years 1 and 3, using the year 1 outcomes as statistical controls. This procedure allowed us to model the 
state of the school during year 3, while adjusting for the initial differences that the schools presented 
during year 1. The first analysis assessed the extent to which the implementation of CSR and individual 
CSR models affected the improvement of social capital among CSR schools, and the reasons for these 
effects. The following is a general representation of the models used to explore RQ 1 for this chapter. 

SocialCapitalYear 04 * SocialCapitalYear 02 .j + R, * CSR _ Model ^ * IMPy 

+ (Covariates) 



— Equation (1) 

A postscript i stands for a school (level 1) and a postscript] stands for a district (level 2). An error term 
Mok is a district-level random error, while the other error term rij is a school-level error. 
SocialCapitalYear04 is either a professional learning community or a collective commitment measure 
derived from the 2004 data. SocialCapitalYearOl is made of the same social capital measure derived from 
the 2002 data. CSR Model is a general representation of a series of dummy variables indicating each of 
the CSR models under examination. Because the reference category is a comparison school, a matrix R; 
hosts a set of coefficients obtained as the difference between the comparison schools and the schools that 
employ each specific CSR model. IMP is a measure of implementation. 
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The second analysis answered RQ 3 by exploring the relationships among social capital, the 
implementation of specific CSR models, and student outcomes in mathematics and reading. The analysis 
for this research question used the same data set and multilevel models as the previous research questions, 
but the outcome measures were different. Unlike the outcomes for RQ 1 , which were school-based mean 
academic achievement outcomes (scores), the changes in academic achievement scores from academic 
year 200 1 to academic year 2002 (spring to spring) were used because scores for 2003 were unavailable. 
The social capital measures from Y ear 1 of the study (academic year 2002) were also used in the analysis. 

For both of the multilevel models, social capital measures such as school means, professional learning 
community, and collective commitment were used as predictors. The CSR models were entered into both 
of the analyses as a series of dummy variables, and the reference category, therefore, was the comparison 
schools. The academic outcomes were measured as school-level mean academic achievement scores in 
reading and mathematics. The following is a general representation of the models used to explore RQ 3 in 
the social capital chapter. 

AchievementChange0203 y -i- * Achievement _ Score * CSR _ Model y + /^ 2 o * IMPy 

+ P 20 * SocialCapitalYear02 y + (Covariates) 




— Equation (2) 

The variable names, parameters, and postscripts mean the same as in equation (1), except for the 
following two variables. AchievementChange0203 is the change in achievement scores between 2002 and 
2003, the subject being either mathematics or reading. Achievement_Score_02 is the same achievement 
scores from 2002. 

Measures 

The first measure of social capital was professional learning community, which measured the degree to 
which teachers were socially connected to one another over professional matters. Activities that would be 
defined as professional learning community activities were taken directly from the surveys distributed 
among the teachers. They included planning lessons or courses with other teachers, diagnosing the needs 
of other students with other teachers, exchanging feedback with other teachers based upon classroom 
observations, acting as a coach or mentor to other teachers or staff in the school (or receiving coaching or 
mentoring), and participating in one or more teaching communities. The response options were presented 
in the surveys in a Likert scale format. Response options included never, a few times a year, once or twice 
a month, once or twice a week, almost daily, and daily. The reliability of the scales for these responses 
was 0.78. 



The second measure of social capital was teachers’ collective commitment. This variable measured the 
level of teachers’ shared commitment to teaching in general. Collective commitment was seen as a 
composite of several activities, which appear in this analysis in the form of variables. Activities measured 
as collective commitment included having a common understanding of the objectives that they were 
trying to achieve with the students, having clear goals and priorities for the school, the extent to which 
most of the teachers had similar values and philosophies of education, the extent to which most 
colleagues shared their beliefs and values about the roles of the central mission of the school, and the 
extent to which most schools shared a forward vision for student learning. Similar to the previous 
measures, the survey questions were presented in the form of a Likert scale, ranging from strongly 
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disagree and disagree to agree and strongly disagree. The reliability of the mean for all of these questions 
was 0.89. 

The analyses in this ehapter used five CSR variables, whieh speeifieally included four individual CSR 
models, and a fifth category for “other,” which consisted of several other models being implemented in a 
few schools. The comparison schools were used as a reference group category, and they simply were 
omitted in the regression models. The analysis also used implementation variables described in chapter 5. 

The final measures used in this analytical model were school mean and academic achievement in 
mathematics and in reading. For academic achievement outcomes, as was discussed in the academic 
achievement chapter, scores were standardized by grade, and so we computed school averages. The 
underlying logic guiding this procedure was that all tests were measuring the same construct, and that the 
performance distributions within the different grades and within districts would be the same. This is 
similar to the steps taken in chapter 7. 

All of our regression models included the following covariates: middle school indicator (dummy coded), 
school size by student enrollment, perceived challenge level by principal (based on principal survey 
questions), percentage minority, percentage free and reduced-price lunch, teachers’ average number of 
years of service, percentage of advanced degree holders among teachers, and average co mm on planning 
time. 

Findings 

Some CSR Models Stimulated Higher Levels of Social Capital 

Results of the first set of analyses reveal that schools that implemented certain CSR models experienced 
higher levels of collective commitment than did their comparison school counterparts. To facilitate the 
interpretation of the coefficients, all of the interval scales were standardized as Z-scores. The analysis 
modeled the level of school-mean social capital. Panel 1 of model 1 in Table 6.1 below presents the 
results of the analysis for the collective commitment measure of social capital. Of the eight models 
included in the overall analysis, two models showed significant results for collective commitment. They 
were ATLAS Communities (SD = .53,/? < .01) and Accelerated Schools Project (ASP) (SD = .72,/? < 

.01). We also controlled for co mm on planning time, so this was not a likely explanation for these 
differences. Moreover, schools that implemented ASP (p = 13), ATLAS Communities (/? = .1 1), and Co- 
nect (/? = .08) experienced higher amounts of social cohesion than their comparison group counterparts. 
These differences, though, were not significant. 

The second research question of the chapter asks the extent to which the implementation of specific CSR 
models affects the level of social capital in schools. This research question hypothesizes that it is possible 
that the very comprehensive nature of CSR itself could be one reason why some CSR schools and models 
overall experience higher amounts of social capital than do their comparison group counterparts. Panel 2 of 
models 1 and 2 tested the extent to which the effects of CSR could be explained by the comprehensiveness 
of the models. Model 1 measures collective commitment and model 2 measures social cohesion. The 
analysis for the research question also added an interaction term that measured the interaction between the 
implementation index and the comprehensiveness index. A school received high ratings on this later 
measure only if it implemented all of the 1 1 CSR implementation criteria uniformly and consistently. 
Testing the interaction between the comprehensiveness index and the overall implementation index helped 
to inform the analysis whether consistent and high levels of implementation did indeed play a role in the 
acquisition of increased and improved social capital ratings or findings. 
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Table 6.1. Results of modeling the relationship between professional learning community and collective commitment by 
school characteristics, model, and selected teacher attributes 
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The results of the analysis revealed that neither the overall level of implementation, nor the 
eomprehensiveness of implementation alone inereased soeial eapital among schools implementing CSR 
models. The analysis therefore concluded that there was no significant relationship between an increase in 
social capital and comprehensiveness among schools implementing CSR models compared to the gains of 
their comparison group counterparts. This makes sense, as both schools were implementing the CSR-like 
practices. 

Specific Modei Components Promote Sociai Capitai 

Overall, two CSR models — ASP and ATLAS Communities — experienced an increase in the two measures of 
social capital (professional learning community and collective commitment). Hence, in some cases, 
externally developed models can increase social capital by developing a collaborative, trusting environment. 
Of further interest, however, is the fact that differences in comprehensiveness did not explain the differences 
in social capital increases between the two groups. 

To probe this issue further, we sought to determine if the structures of the models could explain the lack of 
differences between the two groups. We examined additional sources of information, primarily responses to 
surveys completed by the model developers. Analyses of theses surveys revealed that two models (ATLAS 
Communities and ASP) had the highest ratings in the CSR governance component. These differences were 
statistically significant, and no other models had similar ratings. This is a seemingly logical finding, in light 
of the fact that governance was one of the key descriptions used to measure social capital, especially for 
collective commitment. It is therefore reasonable to conclude that the design of the CSR models is somewhat 
responsible for greater social capital gains for CSR schools implementing ATLAS and ASP. Qualitative data 
also revealed that these two models in particular also stressed the importance of “community development” 
in the schools, which is measured in the form of cohesion. 

Our next step, then, was to analyze the relationship between the implementation of speeific model 
components and social capital, to determine if individual components might explain the 
differences detected in earlier analyses. The implementation index (as described in chapter 5) is 
composed of 10 individual implementation indices, and for this analysis we decided to explore 
each of the components individually, instead of using the overall implementation index. The 
results of this analysis in the form of coefficients are revealed in Table 6.2. The outcomes and 
predictors for this analysis are presented in the form of Z-scores. The results show that variables 
related more closely to teaching tended to be more significantly related to social capital than other 
variables. Some of these variables included, technology use, assessment, pedagogy, inclusion, and 
professional development. The results also revealed that the effects of many of the indices not 
related to teaching were weak. 
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Collective Commitment Is Associated With Achievement Gains in Reading 

Having determined that CSR models may be assoeiated with inereased soeial eapital, we turn to the 
question of student outeomes. That is, does inereased soeial eapital produee student aehievement gains? 
Model 1 assessed the effeets of CSR on aeademie aehievement on both the mathematics and the reading 
test scores. This model added measures of social capital, specifically professional learning community 
and commitment, and further explored the effects of these measures upon mathematics and reading 
scores. Overall, both measures had an effect upon academic achievement. The examination of the effects 
of the measures of social capital upon academic achievement yielded mixed results. Specifically, 
collective commitment had a positive and significant effect upon academic reading (p < .05), it did not 
have a significant effect upon mathematics achievement outcomes (p = 0.75). In reading, collaborative 
commitment once again had a positive and significant effect on the academic achievement outcomes 
ip < .05). Professional learning community had a positive effect upon reading also, but this effect was not 
significant. 

To summarize the results, the implementation of CSR models themselves did not have an effect upon the 
change in academic achievement over the years, but one of the individual measures of social capital had a 
positive and significant effect on academic achievement. The overall analysis, though, did not support the 
hypothesis that the effect of CSR on academic achievement is mediated by social capital. Interestingly, 
collective commitment seemed to have had an independent main effect upon academic achievement for 
both mathematics and reading. Of additional interest is that two CSR models in particular experienced 
greater effects as a result of social capital than did any of the other CSR models. 

Conclusion 

The results of the research questions in this chapter have revealed that some individual CSR models 
experienced greater increases in social capital than did others. Our statistical analyses were also able to 
identify possible reasons for these findings; that is, the CSR models that experienced higher social capital 
were actually designed to improve the measures that we use to define social capital in this paper, such as 
collective commitment and professional learning community. 

The analyses were unable to prove the underlying hypotheses for RQ 3, which did not find any support 
for the idea that social capital may mediate the effects of CSR. Even for this analysis, although we were 
unable to detect CSR effects among the individual models, collective commitment had a positive effect on 
academic achievement in both mathematics and in reading. If we were to summarize the results of the 
findings, we could conclude that if social capital is built into a model design, then it is likely to increase 
or to improve when the model is implemented, and model implementation in turn is likely to promote 
student academic achievement. 
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Chapter 7: Improving Student Achievement 

Yu Zhang 
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Introduction 

The research questions that guide the analyses in this chapter investigate the relationship between the 
implementation of CSR and student achievement. The research questions also explore the relationships 
between implementation and academic achievement. 

The analyses in this chapter investigate not only the effect of CSR on academic achievement, but also the 
extent to which the implementation of a particular CSR model was related to a school’s capacity to 
improve student academic achievement. This chapter also discusses various factors influencing the 
relationship between implementation and student achievement, which lead to decisions to either sustain or 
not sustain the model. In this way, it links the preceding chapter on implementation to the next on 
sustainability. The results of the analyses in this chapter have both research and policy implications. 

This chapter focuses on the first NLECSR research question (described in chapter 1): How effective are 
specific externally developed, research-based CSR models in improving the achievement of all students? 
Broadly speaking, successful CSR model implementation is hypothesized to produce two major changes 
in the CSR schools. First, CSR schools are expected to redesign themselves organizationally upon 
adoption of their externally developed models. Secondly, the restructuring and organizational changes 
resulting from implementation are expected to produce gains in student achievement. The analyses 
conducted in this chapter address the second expected change, which specifically examines the 
relationship between implementation of CSR models in schools and improvement in student academic 
achievement. 



Organization of the Chapter 

Following this introductory section, we describe the methods used, analytic approach taken, and empirical 
results. These sections describe both the effects of CSR on student achievement and attempt to 
disentangle the effects of different components of CSR on achievement. 

Theoretical/Conceptual Framework 

Herman et al. (1999) reviewed 24 CSR models and found that only 3 had rigorous evidence 
demonstrating effects on student achievement. These 3 models were Direct Instruction (DI) and Success 
For All (SFA) for students in grades K-6, and High Schools That Work (HSTW) for students in grades 
9-12. Additional models shown to have marginal effects included Community for Teaming, Different 
Ways of Knowing (DWOK), Expeditionary Feaming/Outward Bound (EFOB), and the School 
Development Program (SDP). 



Consistent with this work, Borman’s recent meta-analysis (Borman et al., 2003) found much the same 
thing. Borman and his colleagues summarized findings from existing studies on the overall effects of 
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CSR, and concluded that the effects of CSR on student achievement were positive overall, and promising. 
They stated that some of the earlier findings may be less reliable because there may have been less rigor 
in the designs, and because most of the evaluations may have been conducted by the developer (Borman 
et ah, 2003). The authors also cautioned that in some cases, evaluations conducted by the CSR developers 
may have yielded estimates of effects higher than evaluations performed by third-party evaluators. 

Finally, the authors caution that based upon the rigor of the design, more rigorous evaluations could 
actually yield more reliable and less significant effects than less rigorous designs. Specifically, the authors 
showed that studies using experimental or quasi-experimental treatment-control comparisons may yield 
effect estimates that are more accurate and usually lower than those of studies based upon analyses of 
CSR pre- and postgains alone. An earlier review of schoolwide reform models (Fashola & Slavin, 1996; 
Slavin & Fashola, 1998) also revealed inconsistent effects of schoolwide reform models upon student 
achievement. 

Several studies have sought to determine the achievement effects of specific CSR models. A 1996 
evaluation of the of the Success For All (SFA) program found that SFA had a significant effect on student 
achievement (Slavin et ah, 1996). Borman et al. (2005) also conducted an evaluation of SFA, and they too 
found that the implementation of SFA had significant effects upon students’ reading scores. Comer 
(1988) conducted an evaluation of the School Development (SDP), and he found that SDP had a positive 
statistically significant effect on student academic achievement. Two independent evaluations of the SDP 
program were conducted both in 1999 and in 2000 (Cook et al., 1999; Cook, Flunt, & Murphy, 2000). 
Flowever, these studies revealed inconsistencies within the SDP program in Chicago (Cook, Flunt, & 
Murphy, 2000) but not in Prince George’s County, MD (Cook et al., 1999). A study by Ross & Lowther 
(2003) also revealed mixed results on the student achievement effects of a single model, in this case, the 
Co-nect model. Another evaluation examined the academic achievement of elementary school students in 
schools adopting the Accelerated Schools Project (ASP) model (Bloom, Flam, Melton, & O’Brien, 2001). 
Results of this evaluation did not present inconsistent results, but the researchers specifically reported that 
after the schools had implemented the model for at least 5 years, third-grade students performed better in 
mathematics and reading compared to the achievement level of third graders prior to adoption of their 
schools’ CSR model. 

Inconsistencies in the studies cited above prompt questions concerning both the relationships between the 
individuals conducting the evaluations and the results. But such explanations are not sufficient — other 
factors are likely to account for differences in the measured effect of CSR on student achievement. The 
next section discusses research studies that have explored a key explanation for differences in results of 
academic achievement among students in CSR schools: the level of implementation of the CSR model. 

Relationship Between Variation in Implementation Level and Student 
Academic Achievement Improvement 

A few studies address the variation in both the level of implementation and consistency of 
implementation across components. Desimone (2002) and Kurki, Aladjem, and Carter (2005) found 
variations in the level of implementation and consistency of implementation among CSR model 
implementing schools. Glennan (1998) and Berends, Kirby, Naftel, & McKelvey (2001) found that some 
schools may experience different challenges and difficulties during the initial adoption and 
implementation processes, thus electing to phase the implementation of CSR at a much slower than 
normal pace. Bodilly (1996) and Smith et al. (1997) offer yet another possible reason for this type of 
variation, suggesting that CSR models can be successfully implemented in schools if the model emphases 
are aligned with priorities in school and district policies. Priority conflicts between CSR implementation 
and school and district policies are likely to lead to problems with regularly scheduled implementation. 
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Fast, Aladjem, Shive, Flerman, and Carter (2001) provided some insight into whieh eomponents of CSR 
were more likely to be implemented. They explained that many CSR models can be distinguished on the 
basis of the centrality of certain key specific components of models and the extent to which they are more 
likely to be implemented in schools. The different models are thus likely to yield differing results 
depending on their areas of emphases and centrality. Finally, Flerman et al. (2000) provided insight about 
patterns of implementation based upon different model structures and emphases. Their analyses revealed 
that structure-based models that focus on instruction or curriculum were more likely to be implemented 
consistently than were philosophy-based models. Taken together, this set of studies provides some insight 
about the realities of the challenges of CSR implementation, and possible reasons for differences in 
results. 

Hypothesized Relationship Between Implementation and Improvement in 
Student Achievement 

The main theoretical underpinning behind the hypotheses presented in this chapter is that two main 
factors influence the achievement of students in schools implementing CSR: time and implementation 
level. The reasoning is that it takes time to implement a CSR model with fidelity, and only when the 
model is implemented faithfully can student achievement be expected to improve significantly. This 
chapter hypothesizes that it will take at least 2 years to reach this level and to reap the academic rewards. 
If the implementation of the model is affected by even some of the factors mentioned, then 
implementation is expected to be interrupted, and this is expected to negatively affect academic 
achievement scores and outcomes. In fact, academic achievement is expected to decline even if 
implementation is fiat and for the initial gains in student achievement to lag even more (or for the initial 
gains in academic achievement to be lower). 

This chapter also hypothesizes that there is a nonlinear relationship between time, implementation of CSR 
models, and academic achievement. The relationship is described as nonlinear because we hypothesize 
that schools will make minimal achievement gains during the first 3 years of implementation, substantial 
gains during years 3-5, and modest gains thereafter.^® What makes this relationship nonlinear, however, is 
that the unidirectional linearity between the three factors (time, implementation, and academic 
achievement) lasts only 2-3 years at most, and then it usually slows down (declines), or even stops after 5 
years of implementation. Thus, the relationship becomes curvilinear as is demonstrated in Figure 7.1. 

This chapter focuses on two specific research questions. 

RQl : To what extent do students enrolled in schools implementing externally developed CSR models 
improve significantly on state- or district-administered mathematics and reading assessments compared to 
their peers in non-CSR comparison schools? 

RQ2 : To what extent is the level of implementation of a CSR model associated with increases in student 
academic achievement performance? 

Methods/Analytical Approach 

This section describes data sources, analyses and measures used, creation of variables, rationales for the 
creation of the variables, specific data used, and analysis-specific caveats as well as technical challenges. 




30 



In this chapter we refer to these three categories of schools as young, middle-aged, and old schools. 
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Figure 7.1. Expected relationship between student achievement and CSR implementation by 
implementation year 




Data 

The data used in this chapter is a subsample of the original sample, which consists of a total of 649 
schools in 2 1 districts in 1 6 states. Data were collected in waves over a total period of 3 years in the form 
of surveys and various forms of standardized academic, statewide administered achievement test scores. 
Although the duration of data collection lasted 3 years, some of the surveys were administered for 2 
years, and others were administered for 3 years. The surveys in the analyses described in this chapter are 
the teacher surveys that were administered in 2002. The implementation data used are described in 
chapter 5. Schools were selected as described in chapter 2. Table 7. 1 shows the distribution of treatment 
schools by model. 

Table 7.1. Number of pairs of scbools matcbed to specific CSR programs, 2002 



CSR Model 2002 


Frequency 


Percentage 


Accelerated Schools Project 


18 


15.6 


ATLAS Communities 


7 


6.1 


Co-nect 


18 


15.7 


Expeditionary Learning/Outward Bound 


6 


5.2 


Modern Red Schoolhouse 


6 


5.2 


Success for All/Roots and Wings 


58 


50.4 


Turning Points 


2 


1.7 


TOTAL 


115 


100.0 



The main school background variables used to match the treatment and comparison schools were students 
eligible to receive free and reduced-price lunch, the population of minority students enrolled in the 
schools, and two additional variables, school size and the average student-teacher ratio. 
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Table 7.2 shows that the CSR and eomparison sehools were similar in terms of the four baekground 
variables, revealing minimal differenees between the two groups. The total number of sehools used in the 
final analyses based upon propensity matching was 115. Attrition in the sample was due to missing data. 



Table 7.2. Descriptive statistics of school background variables for CSR and 
comparison schools 



Variable 


CSR Schools 


Comparison 

Schools 


Difference 




Mean 


SD 


Mean 


SD 


Mean 


Proportion of students eligible for free or 
reduced-price lunch 


0.78 


0.20 


0.80 


0.18 


-0.02 


Proportion of minority students 


0.90 


0.18 


0.92 


0.16 


-0.02 


School size (unit per 1,000 students) 


6.71 


3.83 


6.72 


4.88 


0.00 


Average student-teacher ratio 


16.76 


3.84 


16.77 


4.09 


-0.01 



Note: n= 115. 




The main school-level implementation measures used in these analyses were primarily the teacher 
surveys. The maximum implementation score possible for any component was 1.0, and the minimum was 
zero. The maximum implementation score across models for the treatment schools was 0.89 (pedagogy), 
and the minimum score was 0.53 (the use of technology in instruction) (Table 7.3). The maximum 
implementation score for the comparison schools was .88 (student grouping), and the lowest 
implementation score for this group was 0.53 (use of technology in instruction). Overall, the average 
score for each of the groups was 0.75, thus showing a striking similarity between treatment and 
comparison groups in terms of the implementation measures. 

Table 7.3. Descriptive statistics of implementation measures of the paired schools for 
school-level achievement analyses 



CSR Schools Comparison Schools 



Implementation Indices 


Component 


N 


Mean 


SD 


N 


Mean 


SD 


Influence of assessments/standards 


Assessment 


102 


0.76 


0.06 


103 


0.75 


0.07 


Use of assessments 


Assessment 


108 


0.87 


0.04 


112 


0.86 


0.05 


Curriculum 


Curriculum 


92 


0.82 


0.07 


94 


0.81 


0.07 


Parent/community involvement 


Parental involvement 


97 


0.57 


0.10 


101 


0.58 


0.10 


Implementation of pedagogy 


Pedagogy 


98 


0.89 


0.10 


98 


0.86 


0.12 


Teaching time 


Instruction 


108 


0.84 


0.15 


112 


0.84 


0.15 


Student grouping 


Instruction 


108 


0.87 


0.10 


112 


0.88 


0.07 


Governance 


Governance 


108 


0.67 


0.10 


112 


0.68 


0.09 


Inclusion 


Instruction 


97 


0.73 


0.19 


101 


0.74 


0.18 


Professional development, 
engagement level 


Professional 

development 


102 


0.73 


0.10 


103 


0.72 


0.10 


Professional development, 
type 


Professional 

development 


108 


0.74 


0.14 


112 


0.77 


0.14 
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CSR Schools 


Comparison Schools 


Professional development, 
emphasis 


Professional 

development 


108 


0.70 


0.07 


112 


0.71 


0.08 


Use of technology in instruction 


Instruction 


102 


0.53 


0.16 


110 


0.53 


0.14 


AVERAGE 






0.75 






0.75 





Description of Achievement Data 

School-Level Data 

The initial sample of sehool-level mathematics and reading achievement data were gathered from years 
1999-2003 and consisted of data from 21 districts and 268 schools. The data were aggregated by state at 
the school level. The tests administered varied from district to district, with varying psychometric 
qualities, and the data were reported in various forms. In order to be able to compare the results across 
districts and tests, it was necessary to standardize the results by standardizing the tests across the districts 
and converting them into Z-scores. The tests and their psychometric qualities in their original form are 
presented in appendix Table 7.1. A. Another positive attribute of converting these test scores into Z-scores 
is that although not all the school districts test their children in the same grades, the Z-scores now make it 
possible to generalize the results on a schoolwide level. Aggregating and standardizing the scores at the 
school level also allows one to obtain a more reliable estimate of the school-level academic achievement. 
The grade-level scores used for the students were the average test scores of students in grades 2-5 for the 
elementary school level, and test scores of students in grades 6-8 for middle school students. 

In some cases, more than one type of test scores was reported for each school. When this occurred, the 
best test score, based upon the score reported and the availability of multiple years of scoring was used. 
Examples of different types of scores provided included scale scores, normal curve equivalents, percentile 
ranks, and/or the percentage of students scoring at each proficiency level. Although many of the schools 
provided three different proficiency levels (at, above, and below level), when proficiency levels were 
used in the analyses, the lowest proficiency levels were used in this study, because CSR schools are 
generally composed of students who generally perform at the lowest proficiency levels. 

Table 7.4 reveals that students in CSR schools began with lower test scores than their counterparts in the 
non-CSR schools during the 1998-1999 school year in both reading and mathematics. 

Table 7.4. Average school-level mathematics and reading Z-scores by year 



Subject 


School 


1998- 

99 


1999- 

2000 


2000- 

2001 


2001-02 


2002-03 


Mathematics 


CSR schools 


-0.291 


-0.337 


-0.251 


-0.220 


-0.272 




Comparison schools 


-0.237 


-0.187 


-0.201 


-0.200 


-0.269 




Difference 


-0.054 


-0.150 


-0.050 


-0.019 


-0.004 
















Reading 


CSR schools 


-0.298 


-0.293 


-0.291 


-0.334 


-0.334 




Comparison schools 


-0.249 


-0.207 


-0.254 


-0.103 


-0.254 




Difference 


-0.048 


-0.086 


-0.036 


-0.230 


-0.081 



Note: Difference = CSR - Comparison. 
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This gap was closed somewhat in mathematics, but not in reading. The negative signs next to the scores 
indicate that both groups began below the district average, and they never caught up to the district average 
scores in either reading or mathematics. 

Although the school-level achievement scores were standardized, some analyses of student achievement 
were conducted using student-level data. The next series of tables provides student-level data from five 
districts in five states. 

Student-Level Data 

Student-level achievement data were collected from five districts in this study: Hickoryville, Elm County, 
Dodgeville, Rainfield, and Riverton (the district names are pseudonyms). The tests and years of 
availability varied across districts. Tables 7. 5-7. 9 report the average mathematics and reading scores 
for the CSR and comparison schools included in this study. Consistent with our observations of the 
school-level achievement data, in most districts, the average scores for student-level mathematics and 
reading achievement were observed to be lower for CSR schools than for the comparison schools. In 
terms of the growth, however, CSR schools seemed to have made more progress than their comparison 
schools. The significance of these differences will be tested later by controlling for some background 
variables that could be confounding the results observed in the descriptive tables. 

Hickoryville 

The first district, Hickoryville, is presented in Table 7.5. This table presents the results of mathematics 
and reading performance for students enrolled in schools implementing CSR models in this county (more 
descriptive information is reported in appendix Table 7.2.A). Over the course of the 5 years, although the 
achievement gap was narrowed, schools implementing CSR models did not experience significantly 
greater achievement gains than their matched comparison school counterparts in reading, in terms of the 
scale scores. 

Table 7.5. Mathematics and reading scores for second-grade students 
attending sample schools in Hickoryville, State C, 2000-2001 



Subject 


School 


2000-01 


2001-02 


2002-03 


2003-04 


Math 


CSR schools 


569.6 


598.4 


616.7 


637.8 




Comparison schools 


576.7 


607.9 


626.2 


643.6 




Difference 


-7.1 


-9.5 


-9.5 


-5.7 














Reading 


CSR schools 


572.9 


600.2 


623.4 


635.1 




Comparison schools 


583.1 


608.3 


632.6 


641.2 




Difference 


-10.2 


-8.1 


-9.2 


-6.1 



Note: Difference = CSR - Comparison; n = 18. 



Elm County 



The descriptive information about mathematics and reading scores for students in both CSR and non-CSR 
model schools in Elm County are featured in Table 7.6. The analyses for this population also followed 
students for 3 years (2001-02 to 2003-04, and traced this average for students in grades 3-5. Over the 
course of the 3 years of analysis, the achievement gap was reduced by over 1 0 points in mathematics and 
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over 15 points in reading (more detailed descriptive information is reported in appendix Table 7.3.A). The 
significance of these differences will be tested later by controlling for some background variables. 

Table 7.6. Average mathematics and reading scores for students in grades 2-5 
attending sample schools in Elm County, State A, 2001-02 to 2003-04 







2001-02 


2002-03 


2003-04 


Math 


CSR schools 










Comparison schools 


287.1 


302.1 


313.6 




Difference 


312.6 


323.5 


327.6 






-25.5 


-21.5 


-14.0 


Reading 


CSR schools 


265.7 


293.4 


296.8 




Comparison schools 


296.1 


312.3 


310.3 




Difference 


-30.4 


-19.0 


-13.5 



Note: Difference = CSR - Comparison; n = 10. 



Dodgeland 

The demographic information for Dodgeland are illustrated in Table 7.7. This analysis consisted of 24 
schools, and the mathematics and reading mean scores remained quite similar between the CSR and their 
matched comparison schools over the course of the 5 years of analysis. Although the mathematics score 
increased slightly by eight points, this increase may not be statistically significant. The results of 
significance test on these differences will be reported later by controlling for some background variables 
(more descriptive information is reported in appendix Table 7.4.A). 



Table 7.7. Average mathematics and reading scores for students in grades 3 and 5 
attending sample schools in Dodgeland, State D, 2000-2001 to 2003-04 





Year 


2000 


2001 


2002 


2003 


2004 


Mathematics 


CSR schools 


148.8 


149.5 


152.7 


153.8 


156.6 




Comparison schools 


149.3 


150.8 


151.1 


152.9 


156.3 




Difference 


-0.5 


-1.4 


1.5 


0.9 


0.3 


Reading 


CSR schools 


150.6 


148.9 


150.0 


150.2 


152.4 




Comparison schools 


150.1 


148.6 


150.1 


149.4 


151.8 




Difference 


0.5 


0.4 


0.0 


0.8 


0.6 



Note: Difference = CSR - Comparison; n = 24. 



Rainfield 

Descriptive information for Rainfield is presented in Table 7.8. The Rainfield dataset consisted of 20 
pairs of schools that had implemented a model for between 2 and 7 years in 2002. The results of these 
analyses revealed that Rainfield schools implementing CSR models experienced slightly higher academic 
achievement gains than did their non-CSR model implementing matched comparison counterparts, in 
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terms of the sealed scores. The significance test on these differences will be conducted later by 
controlling for some background variables (see more descriptive information in appendix Table 7. 5. A). 

Table 7.8. Mathematics and reading scores for students in grade 3 attending sample 
schools in Rainfield, State N, 1999-2000 to 2000-2001 







1999-2000 


2000-2001 


2001-02 


Mathematics 


CSR schools 


70.2 


78.5 


79.3 




Comparison schools 


72.7 


78.1 


80.5 




Difference 


-2.6 


0.4 


-1.2 


Reading 


CSR schools 


68.6 


79.9 


80.3 




Comparison schools 


71.6 


79.6 


80.7 




Difference 


-3.0 


0.3 


-0.4 



Note: Difference = CSR - Comparison; n = 40. 

Riverton 

During the baseline year, Riverton schools were slightly ahead of their comparison group counterparts in 
both reading and mathematics, but this difference was not very large in terms of the scale of the test score. 
The gap widened over the years, with students in the treatment schools outscoring students in the matched 
comparison schools during all 3 years of the study. This is reflected in Table 7.9. Significance tests will 
be described later in the chapter (see more descriptive information in appendix Table 7.6.A). 

Table 7.9. Mathematics and reading scores for students in grade 3 attending sample 
schools in Riverton, State M, 2001-02 to 2003-04 







2001-02 


2002-03 


2003-04 


Math 


CSR schools 


587.6 


619.9 


624.0 




Comparison schools 


585.6 


611.7 


612.2 




Difference 


2.0 


8.1 


11.8 


Reading 


CSR schools 


600.9 


627.5 


632.8 




Comparison schools 


592.9 


620.9 


620.8 




Difference 


8.0 


6.6 


12.0 



Note: Difference = CSR - Comparison. Students from 8 schools — 4 CSR and 4 comparison schools. 



Analytical Models/Methods 

This section briefiy explains how the data were prepared for the analyses and reviews the analytic models 
used to answer the research questions. Questions regarding overall analyses of student achievement were 
addressed using time-series models to analyze school-level academic data. Questions regarding student- 
level data were analyzed using both growth modeling and time-series models. 



Data Preparation and Rationale: School- and Student-Level Data 

The achievement analyses were conducted at both the school and student levels. First, for the school-level 
analyses, schools were pooled together across districts. Pooling the data allows the dataset to have a 
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reasonable sample size, whieh in turn provides power neeessary to study and to detect relationships 
between student-level variables, such as the relationship between implementation measures and student 
achievement. 

The student-level data from the five districts provided adequate power for the analyses to explore the 
difference in achievement between students in CSR and comparison schools. These data were not used to 
estimate the impact of school-based implementation variables, as there were very small numbers of 
schools were sampled from each district. Thus, the student-level achievement data were used to respond 
to RQ 1 . 

One important concern when analyzing data from several districts is that different states administer 
different assessments, which are not directly comparable. The ideal solution would be to conduct the 
analyses within the districts, and then possibly to use meta-analyses to summarize these results. This was 
an imperfect strategy for this study, as only a small number of schools were available in each district. 
Hence, although one could use this analytic approach, the study would have to compromise power as a 
result of a small sample size for each analysis. Moreover, for the NLECSR achievement analyses, the 
interest lies in exploring how CSR schools are ranked (by the district) differently from year to year 
relative to their comparison schools, and so it is possible to track these differences by following the 
changes in Z-scores. 

The solution proposed in this case is a matched-pairs design to pool all schools across districts. With a 
matched-pairs design, it is possible to create or to compute a variable that compares the performance of 
CSR students to those of their matched-comparison schools within each district. This newly compiled 
variable computes the difference between CSR and comparison schools for each pair, which makes 
student-level scores more comparable across districts than the Z-score variable. This is because these 
newly computed differences minimize the variances between districts contributed by the various averages 
in Z-scores across districts. Taking the differences between the two matched groups minimizes the 
variances that could be incurred by using data from different districts, such as standardized tests. As the 
analysis for this chapter also seeks to examine the relationship between student academic achievement 
improvement and level of implementation of CSR models, taking the difference between these two 
groups also allows for the comparison of certain implementation practices, especially given that there are 
implementation scores available for both the treatment and the control schools. The next section describes 
the procedures used to rematch or re-pair the groups involved in the analyses. 

Procedures for rematching or re-pairing the groups invoived 

The data preparation and analysis methods described in this section address and also control for some of 
the concerns about differences between the student groups and teaching practices. Although the schools 
were initially matched using percentage minority and free and reduced-price lunch variables, they were 
rematched using propensity scores. Matching schools using propensity scores permits the matching of the 
treatment and the comparison schools using a greater number of background variables. This method also 
allows one to compare the implementation of certain CSR-like practices, given that there are 
implementation scores available for both the treatment and the comparison school groups. For the school- 
level data, once the scores were initially matched using propensity scores, when the differences in 
Z-scores were used as the dependent variables, the averages and differences in school background with 
pairs were used as controls. For the student-level analyses, scaled scores rather than Z-scores were used 
because the matching of the pairs in this study occurs at the school level rather than at the student level. 
Thus, the pairs were maintained at the school level in the analyses, while retaining the quasi-experimental 
design of the study. 
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Methods and Models Used in the Analyses of RQ 1 

RQ 1 asked the extent to which students enrolled in CSR schools improve relative to their peers in non- 
CSR schools. Differences in Z-scores (changes in gaps) in both mathematics and reading between CSR 
and comparison schools were computed and tracked for multiple years. Because the comparison schools 
slightly outperformed the treatment schools at the beginning of the study, a significant reduction in the 
gap between the two groups would indicate that the CSR students were “catching up” to their comparison 
group counterparts, thus implying that the implementation of CSR promotes student academic 
achievement. 

The subquestion about the relationship between student academic achievement and the number of years of 
implementation of CSR in schools was analyzed by separating schools into 1 of 3 categories depending 
upon the number of years of implementation. The groups were divided into groups of 0-2 years, 3-5 
years, and 6 or more years of implementation. Consistent with the life-cycle metaphor, schools 
implementing the model for 0-2 years are described as being “young.” Schools implementing the model 
for 3-5 years are described as being “middle,” and schools implementing the program for more than 5 
years are described as being “old.” The analyses examined which of the three groups experienced the 
largest gains or improvement rates in academic achievements. According to the hypothesis presented in 
Figure 7.1, schools implementing CSR for 3-5 years are expected to experience the largest gains. This is 
because student academic achievement changes are expected to initially lag behind the changes in 
implementation levels, but then are expected to reveal themselves more rapidly between years 3 and 5. 

Methods and Models Used in the Analyses of RQ 2 

Measures of individual components of CSR 

RQ 2 asked the extent to which the level of implementation of specific CSR models is associated with 
improved student achievement. Analyses for this question addressed the relationship between 
improvement in student academic achievement and level of implementation of CSR using two specific 
measures to estimate the level of program implementation. First, we used the individual component 
measures, examining the relationship between the implementation level of each component and student 
achievement. The rationale behind these analyses was that the implementation measures of any or all of 
the CSR components could be either collectively or individually related to student achievement in 
multiple directions. It is possible that some of the components of CSR models could have an effect on 
student academic achievement only when implemented at either high or moderate levels, while others 
may have similar effects when implemented at low levels. It is also possible that for some models, low 
implementation of certain components could have an adverse effect upon student achievement. The 
individual measures approach allowed us to investigate the effects of each individual component at all 
levels. 



Using the average of all measures 

A related set of analyses addressed the relationship between the overall implementation of the 
components of CSR and student academic achievement. For the analysis of RQ 2, it is presumed that the 
relationship between each implementation measure and student academic achievement will vary, 
therefore it will be difficult to reach a general overall conclusion about the level of implementation of a 
CSR model and student academic achievement. With this in mind, it would still be possible to combine 
the multiple measures of implementation into an average, and then to produce one single factor which 
will serve as a predictor. This predictor could help to explain the relationship between implementation 
and student academic achievement. 
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If the analyses were to detect a combined implementation level, then this could indicate that the 
“comprehensiveness” of implementation of all of the components is more important than the individual 
effects of the various components. Comprehensiveness in this case may mean that implementing all of the 
components together is more important than implementing a few components at different levels. 

Implementation of Specific Models and Student Academic Achievement 

To prepare data for these analyses, we created dummy variables for CSR models and the interaction terms 
between the CSR model dummies and implementation measures in the regression analyses. The 
components of these newly created dummy variables explain how average academic achievement 
improvement varies across schools by CSR model, explained by the overall implementation measure. The 
coefficients of these interaction terms explain how the impact of the overall implementation measure may 
vary across schools by CSR model being implemented. 

Basic Statistical Models for Achievement Data 

The statistical models used to answer RQs 1 and 2 included value-added growth and time-series models 
for the individual student-level analyses, and time-series for the school-level analyses. This chapter 
specifically uses two-level time-series regression models, with the first level being the year (level 1 = 
year) and the second level being the school (level 2 = school). The models for the time series analysis are 
described below. 

Time-series model 

Time-level model: 

lij = + A; * Yearn, j + (5^. * YearQl,^ + * 7ear03 .. + 

where: 

rjij is the difference in Z-score between CSR school i and its paired school in year j\ 

Poj indicates the average difference in Z-score in 1 999-2000; 

P ij estimates the difference in the average Z-score between 1 999-2000 and 2000-0 1 ; 

P 2 j estimates the difference in the average Z-score between 1999-2000 and 2001-02; 

Psj estimates the difference in the average Z-score between 1999-2000 and 2002-03; 

YearOly , Year02y , and Year03y are year dummy variables, and standardized by the grand mean and 

standard deviation; and 
Sij is error term. 

School-level model: 

Paj - Zoo + Zoi * ZScore99j -t * Y^eiir _ average j -t * Pair _ difference -f v^j 
where : 

Pxj = ZlO 
Pij ~ y 20 
Phj ~ y 30 

yoo is the average intercept; 

ZScore99j refers to two control variables generated from 1998-99 Z-score. One is the pair-average and 
another is the pair-difference; 
yio, 720 , and yjo are average slopes; 

The variable pair-average refers to the following 4 grand centered variables: 
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SchoolSizeij is the pair-average number of students enrolled in sehool j. 

Pct_Minorityy is the pair-average percentage of minority students enrolled in school j. 

PctFreelunchj is the pair-average percentage of students eligible for free lunch in school j. 

Student _teacher_ratiOij is the pair-average student-teacher ratio in school j. 

The variable pair-difference refers to the following 4 variables: 

SchoolSize Difij is the pair-difference in the average number of students enrolled. 

Pct_Minority_Difij is the pair-difference in the average percentage of minority students. 

Pct Freelunch Difij is the pair-difference in the average percentage of students eligible for free lunch. 
Student teacher ratio Difij is the pair-difference in the average student-teacher ratio. 

Voj is a random error term representing unmeasured factors related to the intercept of the growth curve for 
schools j. 

The model described above is the starting model at school level. More variables are included when 
school-level variables such as such as years of implementation, implementation level and CSR model, and 
their relationship to student achievement are examined. 

Growth model 

Below is the model description for students, beginning in 2nd grade in 2000-0 1 who are followed through 
2003-04 when they were in the 5th grade: 

Note: i = year, j = student and k = school 

Level 1 : time-level model (within student) 

y =77 -fTT ^Pimeye 

Note: Time is coded 2000 = 0, 2001 = 1, 2002 = 2 , and 2003 = 3. 

Level 2: student-level model (within school) 

=Booj^ +Boi 2 t ^ Gendery^Q^^ *ELy'Qf^^i^ * Hispanidr'QQ^ * Blacky'QQ^f^ * Othermimrityy 
Boa *Freelunch-R^^j^ 

Note:. Bqoj. is the average yearly growth from 2000 to 2003. B^^. is the difference in growth rate 
between male and female students. 

Level 3: school- level model 



Bqo/i “ Togo + Tool * CSR + Tqqj Implementation + -I- * Implementation * CSR + 

+ ^003 * Change in implementation y Middle yUg^. 



Note: The most interesting parameters in this model are yon and yon, which measure the impact of CSR 
on the yearly changes in the average achievement score. 



To test the association between CSR age and growth in student achievement, as hypothesized in Figure 1, 
the 3 -year dummy variables were replaced by a time variable at the time-level model and the number of 
implementation years as a predictor at the school-level model was added. 
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The time variable has values of 0, 1 , 2, and 3 to indieate aeademie years 2000, 200 1 , 2002 and 2003 
respeetively. Beeause the hypothesized relationship is similar to the developmental relationship between a 
human being’s age and height, it is positively eorrelated during the early ages, and then unrelated or non- 
linearly related after a eertain age. 

The eonversion of the number of implementation years as a nonlinear predietor was ereated by eonverting 
the error funetion. This funetion gives the probability of a variable when the value of the variable is 
smaller than a value, and then is equal to the integral. The distribution of the eonverted variable looks 
similar to the equation in Figure 7.1. Let x = year of implementation, and then the error distribution of 
implementation year is defined as follows: 

erf{x) = 2/-J^\ dz, z = — — — 

The value of erf(x) eontinually inereases as x inereases and becomes 1 when x is larger than its mean r|. 
By taking erf(x-3)+l, the number of years of implementation was rescaled to be smaller than 1 before the 
3rd year and close to a constant after the 5th year. This transformed years of implementation mimics 
pattern showed in Figure 7.1. The transformed version of number of years of implementation years is 
presented in Table 7.10. 

Table 7.10. Number of years of implementation and its conversion in error function 



Year of implementation (x) 


0 


1 


2 


3 


4 


5 


6 


7 


>7 


Nonlinear conversion, erf(x - 3)+1 


0.000 


0.005 


0.157 


1.000 


1.843 


1.995 


2.000 


2.000 


2.000 



Note: The number of implementation years has a range from 0 to 14, with mean of 4.3 and a standard deviation of 
2.5. 



In the model, the number of implementation years and its transformation are both included in the 
school-level model as predictors. The former predicts a linear relationship and the latter predicts a 
nonlinear relationship. The nonlinear predictor is expected to be a better predictor than the linear one. If 
this is true then it proves that the hypothesized relationship is correct. 

Findings 

CSR Works When Given Time to Work 

The analysis for this question began with 139 pairs of schools (matched by propensity score). Of these 
pairs, 115 had the required 5 years’ worth of data (1999-2003), as is indicated in Table 7.1. During the 
baseline year, students attending CSR schools had lower academic achievement scores than the district 
mean scores. At the end of the 5 years of analysis, the CSR schools had closed the achievement gap in 
mathematics, but not in reading. This trend was bumpy and not consistent from year to year. For instance, 
during some years, the achievement gap either increased or remained the same between the two groups. 
This pattern is illustrated in Figure 7.2. CSR schools received as low as 0.05 points lower in the Z-score 
scales than did the comparison schools in 1999. 
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Figure 7.2. Average difference in Z-scores between CSR and comparison schools 
in mathematics and reading 1999-2003 




When controlling for the school background variables, the estimated change in the achievement 
gap is positive, which indicates that student achievement levels in some CSR model schools 
improved more rapidly than in comparison schools, even though overall the increase is not 
significantly different from zero (see results in Table 7.1 1 for the significance test and appendix 
Table 7.9.A for the estimates). 



Table 7.11. Average difference of school-level mathematics and reading Z-scores by year 
between CSR and comparison schools 



Subject 


School 


Change in Z-Score Gap From 
1999-2000 to 
2002-03 


CSR Effect on Closing Gap 
With Controlling for 1998- 
1999 Z-Score and 
Background Variables 


Math 


Difference between 
CSR and comparison 
schools 




0.146 


0.116 (0.073) 


Reading 


Difference between 
CSR and comparison 
schools 




0.004 


0.090 (0.077) 



Note: Numbers in parentheses are standard error. 



A subanalysis under this research question asked whether the number of years that the model had been 
implemented affected student performance. Figures 7.2 and 7.3 show the change in the difference in 
Z-score between CSR and comparison schools separated by period of years of implementation. According 
to the hypothesis presented for this research question, CSR schools that have implemented CSR models 
for longer periods of time are expected to experience greater gains in student academic achievement than 
schools implementing the models for only a few years. In this study, the number of years of 
implementation of the CSR models reported by schools ranged from 0 to 10. 



The hypothesis for this analysis was that there would be a curvilinear relationship between the number of 
years of implementation of CSR models and academic achievement increases. One of the major findings 
as a result of the analysis was that the rate of increase among the two groups was similar, which could 
lead one to conclude that there is a weak relationship between the number of years of implementation of a 
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CSR model and improvement in reading achievement. Figure 7.3 presents the average difference in 
Z-scores in reading from the 1 999-2000 school year to the 2003-04 school year for schools implementing 
the CSR models by age. 

Figure 7.3. Estimated difference in gap in mathematics Z-scores between CSR and 
comparison schools by the stage of CSR implementation, 1999-2000 to 2002-03 




Figure 7.3 shows that schools with CSR models of “middle” age experienced the smallest achievement 
gaps compared to the comparison groups. During the 2000-0 1 to 2001-02 school years, the improvement 
in academic achievement for schools that had implemented CSR models for 3-5 years was higher than 
that of the matched comparison groups. Flowever, after the initial gains toward the end of “middle age,” 
the rate of improvement for CSR schools began to slow down and eventually vanished in 2002-03. This 
trend ended during the 5th and 6th years, which interestingly also coincided with a time when schools in 
middle age moved into old age. Table 7.12 reports the significance test for estimated difference in the gap 
between middle age and young CSR schools (see the estimates in appendix Table 7.10.A). 

Table 7.12. Estimated differences in gap in the mathematics and reading Z-scores 
between middle stage and young stage for CSR schools, 1999-2000 to 2002-03 





Mathematics in Figure 7.2 


Reading in Figure 7.2 


Difference in Year of 


Estimate 


SE 


Estimate 


SE 


1999-2000 


0.081 


0.172 


0.390** 


0.177 


2000-2001 


0.339* 


0.172 


0.415** 


0.177 


2001-02 


0.270 


0.172 


0.200 


0.177 


2002-03 


0.034 


0.172 


0.101 


0.177 



Note: *p value significant at 0.1 level; **p value significant at 0.05 level; ***p value significant at 0.01 level. 

Figure 7.4 reveals that CSR schools with “middle-aged” models were more successful at closing the 
reading achievement gap in 1999-2000 and 2001-02 (during years 3-4), which corresponds with the 1st 
and 2nd years of implementation for young age group. Schools in the “old age” group experienced similar 
trends in mathematics, as slowly lagging behind in growth compared to their comparison school 
counterparts and then catching up again. One explanation for this pattern could be the adoption of a new 
reform in the old age schools. 



AMKRICAN 
INSTITUTKS 
FOR RESEARCH 



I 




Chapter 7: Improving Student Achievement 



Figure 7.4. Estimated gap in reading Z-scores between CSR and comparison 
schools by stage of CSR implementation, 1999-2000 to 2002-03 




Student-Level Analyses 

We next explore the existenee of possible aehievement differenees between CSR model and non-CSR 
model sehools at the student level. Table 7. 13 reports results of growth models for three districts (see the 
estimates in appendix Tables 7. 1 1 .A-7. 13.A). Table 7. 14 reports the results of the time-series model for 
2 districts (see the estimates in appendix Tables 7.14.A-7.15.A). 



Table 7.13. Per-year rate of CSR school improvement in mathematics and reading, relative to 
comparison schools, 1999-2000 to 2003-04 



Subject 


Hickoryville 2000-2001 
to 2003-04 


Elm County 2001-02 
to 2003-04 


Riverton 2001-02 
to 2003-04 




-0.7 


-0.1 


2.2 


Mathematics 


(1.6) 


(0.97) 


(4.6) 




-0.4 


10.1*** 


1.2 


Reading 


(1.3) 


(0.02) 


(3.1) 



Note: ***p value significant at 0.01 level. 



Table 7.14. Change rate of improvement for CSR schools in mathematics and reading by year, 
relative to comparison schools, 1999-2000 to 2003-04 



Subject 


1999-2000 


2000-2001 


2001-02 


2002-03 


2003-04 


Dodgeland 


Mathematics 


-3.04* 

(0.64) 


-0.39 

(0.64) 


1.84*** 

(0.57) 


1.59*** 

(0.58) 


0.66 

(0.61) 




-2.6* 


0.27 


0.16 


-0.25 


-0.22 


Reading 


(1.68) 


(0.64) 


(0.57) 


(0.57) 


(0.61) 


Rainfield 






-2.9** 


2.8*** 


1.4* 




Mathematics 




(1.4) 


(0.7) 


(0.7) 








-2.9 


3.1*** 


3.2*** 




Reading 




(1.9) 


(0.8) 


(0.9) 





Note: *p value significant at 0.1 level; **p value significant at 0.05 level; ***p value significant at 0.01 level. 








Chapter?: Improving Student Achievement 



■ Hickoryville: Over the course of the 5 years, schools implementing CSR models did not 
experience greater achievement gains than their matched comparison school counterparts in 
reading. 

■ Elm County: Over the course of the 3 years of analysis, CSR schools experienced 
higher/more academic achievement improvement in reading than comparison schools. This 
was not the case for mathematics. 

■ Riverton: Over the course of 3 years, schools implementing CSR models did not experience 
greater achievement gains than their matched comparison school counterparts in reading. 

■ Dodgeland: The schools implementing CSR models improved more academically in 
mathematics than did comparison schools, but not in reading. 

■ Rainileld: Schools implementing CSR models experienced larger academic achievement 
gains in both mathematics and reading than did their non-CSR model matched comparison 
group counterparts. 

The results of student-level achievement analyses are consistent with the finding from the results of 
school-level achievement data. That is, some but not all CSR schools improved at a faster rate than their 
comparison schools. In other words, CSR schools appear, overall, to have no advantage over comparison 
schools in terms of achievement. 

A subhypothesis of RQ 1 predicted differences in student-level scores between the CSR model schools 
and their matched comparison schools as a result of the number of years of CSR implementation. In 
Hickoryville, the findings were consistent with this predicted subhypothesis. Students attending CSR 
schools with “middle-aged” or “old” models performed better than their matched comparison school 
counterparts in the year 200 1-02, but this upward trend ended during the 2003-04 school year. Close 
inspection of the results of the participating schools in Hickoryville reveals that 8 of the 9 schools in the 
study were at the old age, therefore there were no significant improvement trends in academic 
achievement experienced, once again, just as the hypothesis had predicted. The analysis also revealed that 
the relationship actually became weaker during the later years. 

Table 7. 15 lists the distribution of CSR schools in the five districts by their CSR age during the 2001-02 
school year. It presents partial explanations for some of the variation in student academic performance in 
CSR schools displayed in Tables 7.13 and 7.14. 

Table 7. 15 reveals that if the predicted relationship between CSR age and improvement in student 
achievement from the school-level data is correct, in the district in which no significant CSR impact on 
student achievement was found, most CSR schools tended to be of old age. This is evident in districts like 
Hickoryville and Riverton, in which the CSR models were of old age during the 2001-02 school year. 
Some interesting findings in Figures 7.3 and 7.4 are that older CSR schools begin to make academic 
achievement gains during years 8-9 of implementation. This could certainly be the case for Rainfield, 
where six of the CSR schools were in the 9th year of CSR implementation in 2001-02 school year. 
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Table 7.15. Number of CSR schools from five districts included in student-level analyses, by 
year of CSR implementation, 2001-02 



District 


Years of Impiementation 


1 


2 


3 


4 


5 


6 


7 


8 


>8 


Hickoryville 










1 


6 


1 


1 




Elm County 






4 


1 












Riverton 




1 




2 




6 


1 






Dodgeland 






1 


6 




1 


3 


1 




Rainfield 


3 


2 


2 


1 


1 




5 




6 



CSR Works When Implemented With Fidelity 

In this analysis, two additional variables were created and used to answer the research question. The first 
variable created was the average implementation level variable for CSR schools. This variable was 
created strictly for information purposes, and not for comparison. It measures the level of implementation 
of CSR-like activities in both treatment and control schools. The second variable created measures the 
difference in implementation between the CSR model schools and their comparison school counterparts 
in each pair. This variable measures the similarity in implementation level between the model adopting 
schools and their non-model-adopting counterparts in the pairs. 

Table 7. 16 shows the relationship between the implementation of specific components of CSR and 
academic achievement improvement based upon school-level data (see appendix Table 7.16.A for the 
results of the estimates of type of professional development on student academic achievement). Very few 
individual implementation components revealed a significant relationship between level and student 
achievement. Even the direction of the relationship was not always positive. 

This means that a higher level of implementation of an individual component does not necessarily 
guarantee greater improvement in student achievement than in comparison schools, because some 
comparison schools might also be implementing CSR-like practices. When CSR schools implemented 
some individual components, for example parental involvement, the CSR schools experienced greater 
improvement in student achievement than their comparison schools. Of the two new variables created, the 
average improvement score was found to be better at predicting academic achievement gains in reading 
than in mathematics. One possible explanation for this is that there was a large number of SEA schools in 
the sample and their main area of focus is reading. In general, however, the relationship between the CSR 
level of implementation for individual components and improvement in student achievement is unclear 
and inconclusive. 



Relationship Between Uniformity of CSR Component Implementation and Student 
Academic Achievement 



In order to examine the overall relationship between CSR implementation and improvement in student 
achievement, new variables were created by taking the average of all individual implementation 
measures. This newly created implementation score computes the average of all of the implementation 
measures reported by each CSR school, and it predicts growth or gains in student achievement. When a 
CSR school implements some but not all components at a high level, the average implementation score is 
not expected to improve much. Table 7.17 reports the results of this newly created measure. Table 7.17 
reveals that the average implementation measure that was created weakly predicted academic 
improvement in mathematics and reading. 
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Table 7.16. Differences in implementation levels predicting differences in mathematics and 
reading Z-Scores between CSR and comparison schools 





Model for Mathematics Achievement 


Model for Reading Achievement 




Model for Predicting Mathematics Score 


Model for Predicting Reading Score 




Implementation 
Score of CSR 
Schoois 


Difference in 
Implementation 
Score Between CSR 
and Comparison 
Schools 


Implementation 
Score of CSR 
Schoois 


Difference in 
Impiementation 
Score Between 
CSR and 
Comparison 
Schools 


Professional development (type) 


0.11 


(0.656) 


0.59 


(0.470) 


-0.24 


(0.754) 


1.06* 


(0.541) 


Professional development 
(engaging) 


0.11 


(0.970) 


0.10 


(0.675) 


0.09 


(1.123) 


0.85 


(0.783) 


Professional development 
(emphasis) 


-0.13 


(1.243) 


0.16 


(0.796) 


0.78 


(1.449) 


0.20 


(0.925) 


Organization and governance 


-0.82 


(1.009) 


-0.03 


(0.761) 


-0.08 


(1.187) 


0.09 


(0.894) 


Curriculum 


-0.18 


(1.576) 


1.23 


(1.200) 


1.68 


(1.755) 


0.58 


(1.334) 


Pedagogy 


-0.51 


(0.851) 


-0.38 


(0.649) 


-0.19 


(0.992) 


-0.26 


(0.735) 


Grouping strategies 


0.85 


(1.195) 


-0.89 


(0.929) 


0.71 


(1.369) 


-1.18 


(1.053) 


Inclusive strategies 


0.10 


(0.515) 


-0.43 


(0.477) 


0.53 


(0.606) 


-0.80 


(0.554) 


Scheduling 


-0.31 


(0.604) 


-0.64 


(0.465) 


0.33 


(0.701) 


-1.20** 


(0.541) 


Assessment 


2.94 


(2.280) 


0.53 


(1.314) 


-0.77 


(2.646) 


3.46** 


(1.479) 


Parental involvement 


-0.86 


(0.881) 


1.97“* 


(0.773) 


-1.35 


(1.048) 


2.06** 


(0.919) 


Technology 


-0.52 


(0.568) 


0.55 


(0.481) 


-0.61 


(0.668) 


0.37 


(0.566) 



Note: Numbers in parentheses are standard errors. *p value significant at 0.1 level; **p value significant at 0.05 level; 
***p value significant at 0.01 level. 

One possible explanation for this weak relationship between average implementation score and 
achievement gains is that some schools may implement some CSR model components, and thus may 
receive scores for implementing something rather than doing nothing. It is also possible that similar levels 
of average implementation of components between comparison and treatment schools may have resulted 
from various scenarios of implementation. 

To elaborate, the standard deviation of, for example, the average implementation level is significantly 
associated with academic achievement improvement in reading. The sign of the coefficient is positive 
because schools with a high average tended to have a large variation among components. However, 
because a large standard deviation indicates a low level of comprehensiveness or uniformity in 
implementation, as was expected, a low level of comprehensiveness or uniformity should be associated 
with an average level of implementation. The sign of the interaction term between the average level and 
standard deviation is negative and significant, which suggests that the level of comprehensiveness or 
uniformity might be inconsequential when the average implementation level is low, as is shown by the 
positive coefficient of comprehensive measure. When the average level of implementation level increases, 
the comprehensiveness of implementation becomes crucial, because a low level of comprehensiveness 
indicates that not all components have been implemented at the same high level, and it is difficult for a 
CSR school to have high fidelity. The negative coefficient of the interaction term may suggest a negative 
impact of lack of comprehensiveness when the average level of implementation is high. This finding 
brings to light the importance of comprehensiveness or uniformity. These findings are displayed in 
table 7.17. 
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Table 7.17. Relationship between the average of 2002 implementation scores and the 
differences in mathematics and reading Z-scores between CSR schools and comparison 
schools, 2000-03 





Model for Math 


Model for Reading 


Intercept 


-7.079* 


3.789 


-8.361* 


4.680 


Time 


0.034 


0.035 


-0.023 


0.031 


Average of 2002 implementation score of CSR school 


8.833* 


4.970 


10.333* 


6.143 


Average of difference in 2002 implementation score between 
CSR and comparison school 


1.173 


1.714 


0.515 


2.032 


Standard deviation of 2002 implementation score of CSR 
schools 


49.193 


30.239 


84.374** 


37.382 


Average of difference in standard deviation of 2002 
implementation score between CSR and comparison school 


1.844 


2.407 


0.273 


2.903 


Interaction: average with standard deviation of 2002 
implementation score 


-64.446 


42.352 


-108.440** 


52.329 


ATLAS Communities 


-46.624 


40.250 


-28.779 


49.884 


Accelerated Schools Project 


-6.916 


5.544 


-17.475** 


6.857 


Co-nect 


-3.447 


4.703 


-10.628* 


5.813 


Other 


-11.360** 


4.661 


-17.916*** 


5.761 


Interaction: ATLAS with average of 2002 implementation score 


63.358 


54.291 


38.428 


67.281 


Interaction: ASP with average of 2002 implementation score 


9.926 


7.663 


24.200** 


9.479 


Interaction: Co-nect with average of 2002 implementation score 


5.197 


6.348 


14.768* 


7.846 


Interaction: other CSR models with average of 2002 
implementation score 


15.648** 


6.142 


24.262*** 


7.591 


Z-score difference in 1999 


0.603*** 


0.077 


0.518*** 


0.095 


Average Z-score in 1 999 


-0.017 


0.075 


0.046 


0.093 


Average percentage of students who were eligible for free or 
reduced-price lunch 


0.020 


0.085 


0.108 


0.105 


Average percentage of minority students 


-0.078 


0.074 


-0.049 


0.091 


Average school size (unit per 1,000 students) 


-0.092 


0.084 


-0.111 


0.104 


Average student-teacher ratio (unit of 10 students per teacher) 


0.088 


0.080 


0.179* 


0.098 


Average difference in percentage of students who were eligible 
for free or reduced-price lunch 


0.249 


0.392 


-0.182 


0.484 


Average difference in percentage of minority students 


-0.612 


0.413 


-0.324 


0.511 


Average difference in school size (unit per 1 ,000 students) 


-0.013 


0.019 


-0.012 


0.023 


Average difference in student-teacher ratio (unit of 10 students 
per teacher) 


0.037 


0.247 


0.543* 


0.306 


Variance 










Intercept, school-level 


0.235*** 


0.079 


0.436*** 


0.107 


Covariance 


-0.017 


0.030 


-0.009 


0.031 


Year of implementation 


0.053*** 


0.019 


0.035** 


0.016 


Time-level 


0.359*** 


0.035 


0.333*** 


0.032 



Note: *p value significant at 0.1 level; **p value significant at 0.05 level; ***p value significant at 0.01 level. 
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Relationship Between Individual CSR Models, Their Comparison Schools, and Student 
Academic Outcomes 

Success for All/Roots and Wings stood out among our sample as demonstrating improved student 
achievement. Table 7.17 displays these results. In Table 7.17, SFA is the reference category.^' Negative 
coefficients mean that ASP, ATLAS Communities, and Co-nect experienced smaller increases in reading 
scores than did the SFA model schools relative to their comparison schools. Overall, the SFA model 
schools showed higher implementation scores than did the comparison schools, as is depicted by the 
negative and significant coefficients of CSR model dummy variables. However, the positive and 
significant coefficient of the interaction terms between the average implementation level and CSR 
dummy variables reveal that the effect of implementation level was higher for predicting reading success 
among non-SFA schools than among SFA schools. In other words, if ASP, ATLAS Communities, and 
Co-nect schools implemented their models at the same levels as SFA schools, then the coefficients 
predicted greater gains in schools implementing these respective models than did schools implementing 
SFA. Thus, the relationship between implementation level and improvement in academic achievement in 
reading was far stronger for schools implementing ASP, ATLAS Communities, Co-nect, and other CSR 
models. 

Conclusion 

This chapter explored the relationship between CSR model implementation and student achievement by 
examining level of implementation, number of years of implementation, and academic achievement. The 
analyses yield four major conclusions. 

There is no apparent difference in the performance of students in CSR schools and comparison 
schools. 

Close examination of the data reveals that some CSR schools even experienced smaller increases in 
academic achievement than their comparison schools. This analysis, however, masks underlying 
relationships between implementation of CSR and student achievement by grouping all models together 
into a single hypothetical CSR effect. 

However, CSR works when given time for implementation. 

The relationship between CSR implementation and student achievement varied by the number of years of 
CSR model implementation. Schools that have implemented CSR for 3-5 years were more likely to 
experience statistically significant associations between implementation and achievement. 

Also, CSR works when implemented with fidelity. 

Our analyses further suggest a significant relationship between overall fidelity of implementation and 
improvement in student achievement. The analyses revealed that a high level of implementation was 
associated with high improvement in achievement. 



This analysis is limited to four specific CSR models (and a category for others) that have enough schools within 
their model groups in the sample to allow for reliable estimates of model group effects. 









r 


6 


IUIb 


AMKRICAN 

INSTITUTES 1 

1 OR researc;h ■ 


I 

i 




Chapter?: Improving Student Achievement 



SFA, in particular, works. Evidence suggests other models work may work as well. 

Of the models speeifically tested, SFA sehools showed statistically significant relationships between 
implementation and achievement. Three models, however, displayed the potential for similar evidence, 
were they implemented as highly as SFA. In fact, the interaction between implementation and 
achievement for ASP, ATLAS Communities, and Co-nect was stronger than for SFA, suggesting that 
these models would see even greater student achievement gains than SFA schools if they were as highly 
implemented. 
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Chapter 8: Sustainability 

James E. Taylor 



Introduction 

The preceding chapters have explored CSR through its initiation phase, implementation process, and 
relationship to student achievement and social capital. This chapter concludes our story by examining 
whether schools sustain or discontinue their formal affiliation with CSR model providers and, more 
importantly, whether schools sustain or discontinue implementation of the practices espoused by the CSR 
model providers with whom they worked. 

Understanding the history of unsustained reform in the U.S. educational system, the architects of CSR 
attempted to design more complete reform models that were less easily displaced from the schools 
because they intervened in a coordinated way on a broad set of components of the school. As evidence of 
the importance CSR developers and researchers placed on sustainable reform, they cautioned at various 
points in the early research on CSR that CSR models needed time, often specified as a 3- or 5-year 
sustained period, to exhibit improvement in student achievement among other outcomes (e.g., Bodilly, 
1998; Slavin & Madden, 2000). Also understanding the importance of ensuring that reform efforts were 
sustained, the CSR legislation and regulations made it clear that the 3-year CSR grants were intended as 
seed money to help schools sustain reform during those 3 years and to develop a plan to make the reform 
self-sustaining beyond the duration of the grant (No Child Left Behind Act, 2002). Despite these 
intentions, it has become clear over the course of our work on NLECSR that one of the greatest 
challenges, if not the greatest challenge, to comprehensive school reformers is sustaining reform over a 
period long enough to produce substantial effects. 

There are many ways of thinking about the end of the life cycle of CSR. We posit eight concluding 
scenarios to define the end of a CSR effort: 



1 . Nonreform never takes hold and ends quickly. 

2. Momentary reform flourishes briefly but quickly dies or is overtaken by another reform. 

3. Nominal reform establishes itself but in name only and is eventually abandoned. 

4. Resident reform establishes itself but persists in name only. 

5. Transient reform establishes itself, changes the system, and then passes away leaving little 
evidence that it ever occurred. 

6. Temporary reform establishes itself, changes the system, but gradually gives way to the forces of 
inertia and persists in name only. 



7 . 



Sustained implementation is sustained and overtakes whatever preceded it so completely that it is 
institutionalized as the status quo and ceases to be “reform.” 
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8. Sustained implementation within a single sustained reform effort is sustained and achieves a 
dynamic equilibrium, making continual adjustments to fit the needs of a continually changing 
environment. 

These eight scenarios fit four pairs. The pairings represent key theoretical and substantive differences 
between pairs and minor differences within pairs. Figure 8.1 displays these pairs and the alternative life 
cycle trajectories they represent. 

Figure 8.1. Reform trajectories 




Years since adoption of reform 



The first pair (scenarios I and 2) represents reform efforts that lasted only briefly (i.e., less than 3 years), 
whereas the remaining three pairs all require at least this minimal degree of longevity. The second pair 
(scenarios 3 and 4) represents superficial implementation, in which schools name their program but 
exhibit little evidence of the practices associated with that program. The only difference between the 
scenarios in this pair is whether the name of the reform effort was eventually dropped or retained. In 
contrast, the third pair (scenarios 5 and 6) exhibits substantial implementation of the practices related to 
their model at one time, but have since abandoned many or most of those practices. Finally, sustained 
reform is found only in the fourth pair (7 and 8), where the practices of the reform program remain clearly 
evident. 

This final pair represents truly sustained CSR efforts, in which CSR practices persist over time. Sustained 
implementation (scenario 7) represents schools in which CSR persists over time and overtakes whatever 
preceded it so completely that it is institutionalized as the status quo and ceases to be “reform.” Sustained 
implementation within a single sustained reform effort (scenario 8) is sustained and achieves a dynamic 
equilibrium, making continual adjustments to fit the needs of a continually changing environment. The 
distinction between the last two scenarios is subtle and will be clarified further in this chapter. The key 
distinction is that schools that fall into scenario 7 dissolve their formal relationship with their CSR model 
provider (potentially even formally adopting another model), but they sustain the practices they learned 
from that CSR program or model because those practices have become taken for granted, internalized, or 
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ingrained in the life of the school. Schools that fall into scenario 8 maintain their same formal CSR effort, 
but within it they take a more flexible approach to continuously improve and adapt to meet environmental 
demands. (Consequently, they end up with lower measures of implementation fidelity.) 

The school reform literature (e.g., Tyack & Cuban, 1995) points to very few examples of scenarios 7 and 
8, truly sustained reform. For illustrative purposes. Figure 8.2 displays how the schools in this study are 
distributed into each of these 8 scenarios. 

Figure 8.2. Concluding scenarios for CSR 




The two darkest slices on the left of the pie chart representing scenarios 7 and 8 reflect the fact that over 
one third of the schools in this sample managed to achieve this high level of sustainability. The remaining 
two thirds of the pie indicate some form of unsustained reform. 

Before we proceed, it is also reasonable to attempt to define the scope of the problem of unsustained CSR 
relationships both in previous research and in this study. Berends, Kirby, Naftel, and McKelvey (2001) 
note that at least 24% of the schools in their study ended their relationship with their CSR developer. In a 
study of one urban district, Datnow (2001) reports that after 3 years, reforms expired in 6 of the 13 
schools under study. Although a low response rate may bias results. Academic Information Management, 
Inc. (AIM; 2003) finds that 25% (27 of 106) of the responding schools had discontinued their original 
CSR model by approximately the 5th year after adoption. Evans, Baugh, Sheffer, Martin, and Scarentino 
(2004) find that 36% (28 of 78) of the Pennsylvania schools visited 5 years after adoption were no longer 
implementing the CSR model. To be clear, each of these percentage rates represents the prevalence of 
unsustained reform relationships (rather than the prevalence of unsustained implementation of a reform, 
for which no previous empirical estimates of prevalence were found). 



’ As noted below, the analyses in this chapter draw only on treatment schools with full implementation data. 
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Earlier we indicated (Figure 8.2) that a third of the schools in this sample achieved sustained reform. A 
more fundamental question is how to define the scope of the problem of unsustained CSR relationships 
both in prior research and in this study. Table 8.1 illustrates the scope of the problem in this study. 

Table 8.1. Number and percentage of schools that dropped or switched their relationship with 
a CSR model provider (summary table) 



A/= 395 
Schools 


Dropped or Switched 
Relationship With CSR Model 


Dropped Relationship 
With CSR Model 


Switched Relationship to 
Another CSR Model 




N 


% 


N 


% 


N 


% 


2002-03 


73 


18.5 


51 


12.9 


22 


5.6 


2003-04* 


77 


21.3 


54 


14.7 


25 


6.8 


2002- 03 or 

2003- 04 


130 


32.9 


92 


23.3 


38 


9.6 



Note: The total number of schools is less than 395 for 2003-04 percentages because 33 schools are missing on the 
dropped relationship variable and 26 are missing on the switched relationship variable. 



Table 8. 1 shows that nearly one third of the CSR schools in our sample dropped their affiliation. 
Consequently, two thirds of the schools sustained their affiliation. Without any context for these data, it is 
difficult to understand their significance (if any). The schools that sustained their affiliation may not be 
implementing many of the practices espoused by the reform developer. We should, therefore, differentiate 
between a sustained relationship and sustained implementation. 

Organization of the Chapter 

For the purpose of exploring the relationship between implementation and sustainability, this chapter 
investigates the nature of the supporting conditions and strategies that are necessary to effectively 
implement and sustain CSR at the school and district levels. Although some studies may often treat 
discontinuation of the treatment as a nuisance rather than as the subject of research, this chapter explores 
the various conditions that lead to the different scenarios of sustainability and discontinuation as 
presented in the overview. 

This chapter begins with the proposition that before one can research sustainability one must be clear 
about what is being sustained. It continues by examining the different definitions of sustainability found 
in the literature. The chapter then reviews the literature on sustained reform and provides a conceptual 
framework for examining sustained reform. Using this framework, the chapter then presents the results of 
two analyses that take advantage of the longitudinal quantitative data from a large sample of CSR 
schools. The first analysis examines whether there are certain factors that explain why schools drop or 
switch a model. The second analysis examines sustainability as it relates to whether schools can drop their 
relationship to a model provider, but continue to engage in practices that closely resemble those 
developed and implemented as part of their former CSR models. The chapter concludes with policy 
recommendations based on our findings. 

Primary Research Question 

This chapter primarily addresses the third NFECSR research question: 

♦ RQ 3 : What supporting conditions and strategies are necessary to effectively implement and 
sustain CSR models in schools and school districts? 



i 
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Chapter-Specific Research Questions 

To address this question, the ehapter uses both quantitative data and anaiyses and quaiitative ease study 
data to answer two more speeifie questions: 

♦ What faetors are assoeiated with a higher iikeiihood of sehoois’ sustaining their reiationships with 
CSR modei providers? 

♦ To what extent does modei impiementation cease after a schooi formaiiy drops its CSR modei or 
to what extent does it persist as a resuit of prior impiementation? 

Theoretical/Conceptual Framework 

Defining What Is Sustained: Reform Affiliation or Reform Implementation 

The expioration of the reiationship between impiementation and sustaining reiationships with CSR modei 
providers requires the definition of severai key terms. Readers wiii remember the definition of 
impiementation as fideiity from chapter 5 of this report. Impiementation is measured here as the extent to 
which the CSR program is deiivered to the intended recipients in the intended way. The other key 
definition concerns what it means to have a sustained reform effort and what distinguishes a sustained 
reform effort from sustained impiementation of a reform: 

♦ A sustained reform relationship is defined as the continuation of the format reiationship between 
a schooi and its extemai CSR modei provider over a number of years. 

♦ A sustained implementation of reform is defined as consistentiy high ieveis of fideiity of 
impiementation of the CSR modei over a number of years. 

We reviewed the research iiterature on sustained reform as there is a wide divergence of approaches and 
definitions. Nonetheiess, there are muitipie exampies of uses of both ideas — sustained reform reiationships 
and sustained impiementation — though few attempts, untii now, to discriminate between the two. 

Severai empiricai studies of CSR have confronted the fact that sehoois drop their format affiiiations with 
CSR modei providers, or what we term here, unsustained reform relationships. For exampie, in a five- 
state study for the Southwest Educationai Deveiopment Laboratory (SEDL), Academic Information 
Management, Inc. (AIM, 2003) examines “program continuation” measured by whether the school 
indicated it had discontinued its original CSR model. Similarly, Datnow (2001) uses the terms 
“expiration,” “dropped reform model,” and “abandoned reforms” to describe instances where schools did 
not sustain a specific formal contractual affiliation with an extemai CSR model provider. Importantly, she 
distinguishes the six schools that dropped their formal reform affiliation but may have continued working 
on the reform effort on their own, from two schools where respondents “never admitted to fully 
abandoning CES, but our observations and interviews confirm that the reform was virtually absent in 
practice” (p. 18). Often there seems to be an implicit (and incorrect) assumption that once these formal 
relationships end, so does the implementation of the practices associated with the reform. 



In contrast, several conceptual pieces on CSR sustainability stress the idea of sustained implementation of 
a reform in terms of the stable use of reform-related practices over time. Datnow (2001) provides a 
summary of the literature that sees sustained reform as the institutionalization, or stable, taken- for-granted 
use, of reform-related practices where those practices become fully internalized and part of how the 
school does business. 
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When one speaks of the sustainability of a reform, one is typieally interested in knowing 
whether the reform lasts over time and beeomes an institutionalized feature of a school. 

Whereas as newer studies use the term “sustainability” (e.g., Hargreaves & Fink, 2000; 

Yonezawa & Stringfield, 2000; Datnow, Hubbard, & Mehan, 2002; Datnow & Stringfield, 

2000), earlier studies discuss “institutionalization” (e.g., Anderson & Stiegelbauer, 1994; 

Berman & McLaughlin, 1978; Cuban, 1986; Tyack & Tobin, 1994). Although in 
dictionary terms, sustainability refers to longevity and institutionalization refers to 
something becoming an established practice, their definitions in the research literature are 
inextricably connected. For a reform to be sustained, it must become institutionalized. So 
too, when a reform is institutionalized, it has been sustained over time. (p. 4) 

Most empirical longitudinal studies of CSR implementation do not explicitly examine sustainability but in 
fact are based implicitly on the idea of sustained implementation of a reform (e.g., Berends, Kirby, Naftel, 
& McKelvey, 2001; Kurki & Aladjem, 2005). 

We are not aware of any empirical studies that examine how dropping a reform relationship is associated 
with longitudinal change in the implementation of that reform. The AIM (2003) study asked schools that 
had discontinued their reform relationship whether they had attained high levels of implementation before 
dropping the reform, finding that nearly half of those schools reported having reached the stage of 
“institutionalization” before they dropped. If correctly understood by respondents, this would appear to 
indicate that implementation of many reform-based practices persisted after they dissolved their 
relationship with the reform developer, but no information was collected about their current level of 
implementation after the schools dropped the reform relationship. Another study by Evans, Baugh, 
Sheffer, Martin, and Scarentino (2004) measured implementation in schools that had discontinued use of 
their CSR model, but longitudinal data were not collected. 

As with “sustained reform,” “reform implementation” has had many different meanings. As discussed in 
chapter 5 (see also Aladjem, 2003), we measure implementation as fidelity: the extent to which the 
program or treatment of interest is delivered to the intended recipients in the intended way. The approach 
described in chapter 5 is based on the idea that to measure the fidelity of implementation, we should 
measure the levels of schools’ and teachers’ activities and compare those against the levels of those 
practices that CSR program developers report to be “full” implementation. Thus, the challenge of 
measuring implementation as fidelity is finding the difference between the positive, empirical reality of 
school life and the normative vision of CSR developers. 

Sustaining CSR Modeis: Factors From the Literature 

To understand what factors may make schools more likely to sustain their CSR efforts, we turn to the 
relevant literature. The key question is what makes schools likely to switch or drop their model. Prior 
research suggests that sustainability depends on a wide array of variables. We have organized these 
variables into 1 1 categories: 

1. High local school capacity (e.g., Stringfield, 1998; Reynolds, Stringfield, Creemers, & Teddlie, in 
press; Florian, 2000) 

2. Supportive political context (Bodilly, 1998; Yonezawa & Stringfield, 2000; Florian, 2000; 
Berends, Kirby, Naftel, & McKelvey, 2001; Datnow, 2001) 

3. Sufficient funding (Berends, Kirby, Naftel, & McKelvey, 2001; AIM, 2003; Evans, Baugh, 
Sheffer, Martin, & Scarentino, 2004) 
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4. Positive student outeomes (Yonezawa & Stringfield, 2000; AIM, 2003) 

5. Fit or alignment between the reform design and the school (Datnow & Stringfield, 2000; AIM, 
2003) 

6. Instructional leadership and leadership stability (Bodilly, 1998; Florian, 2000) 

7. Faculty retention (Hargreaves & Fink, 2000) 

8. Faculty commitment (Moffett, 2000; AIM, 2003), including factors associated with initial buy-in 
and the reform adoption process (Datnow, 2000) 

9. Practical concrete reform specifications that are structured into the daily life of the school 
(Yonezawa & Stringfield, 2000; Florian, 2000) 

10. Sustained professional development (Yonezawa & Stringfield, 2000; Moffett, 2000; Florian, 

2000) 

11. Model developer assistance (Berends, Kirby, Naftel, & McKelvey, 2001) 

Research on effective schools (e.g., Stringfield, 1998; see also Datnow & Stringfield, 2000) has found 
that “positive outlier” schools, that is, unusually high achieving schools given their degree of social 
economic disadvantage, can sustain their improvement efforts over a decade or more. Unfortunately, 
these relatively higher capacity disadvantaged schools are less in need of reform than their lower capacity 
disadvantaged peers. Nevertheless, high capacity within a school is likely to be strongly related to 
whether schools can sustain their reform efforts. Berends, Kirby, Naftel, and McKelvey (2001) conducted 
“exit interviews” with 30 principals whose schools dropped their CSR model, finding that lack of 
funding, lack of district and state support, and dissatisfaction with the assistance from CSR model 
developers were the primary reasons schools dropped their affiliation with their CSR model. Yonezawa 
and Stringfield (2000) found that schools sustained reform when there was political support, alignment of 
the “cultural logic” of the reform design and that of the local reformers, and when reform was structured 
into the daily lives of the school community. Datnow’s (2001) case studies identified the same three 
general factors. Datnow cites that additional sustainability factors are quite predictable and include such 
things as genuine interest in change, teacher and administrator support, a critical mass involved in 
implementation, sustained professional development, and a practical plan for implementation and 
monitoring of the change effort. Florian (2000) identifies five factors to which staff members in 
sustaining schools attribute their ability to sustain reform practices: ongoing use of reform practices, a 
culture of learning and innovation, support structures, leadership, and political context. Hargreaves and 
Fink (2000) report that succession in leadership and retention of staff also influence the continuation of 
instructional improvement. AIM’s (2003) follow-up study of schools after their CSR funding had ended 
found that the most important reason for the CSR program remaining in place was successful student 
outcomes. Respondents reported three additional main reasons for continuation: alignment with school 
goals, continued funding, and strong support by teachers and administrators. In a similar postfunding, 
follow-up study, Evans, Baugh, Sheffer, Martin, and Scarentino (2004) found that the most frequently 
cited cause for discontinuation was that district mandates regarding the specific scope and sequence to be 
followed were incompatible with the existing CSR model; the second most cited cause was lack of 
funding. In summary, existing research has identified a series of plausible sustainability factors, but has 
not yet proven consistent linkages between these factors and sustainability. 



Sustaining CSR Modeis: Factors From NLECSR Data 

While these factors address sustainability, they do not address the phenomenon of schools’ dropping CSR 
models per se. It is important to explore the issue of schools’ dropping their models both qualitatively and 
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quantitatively. To gain a qualitative understanding of the reasons for whieh sehools drop CSR models, we 
drew upon data from prineipals and teaehers. These data ineluded open-ended responses from the 
prineipal surveys as well as transcripts of structured interviews with principals and teachers. Qualitative 
data were used to refine the model of the factors affecting schools’ decisions to sustain or drop their CSR 
model used as part of the quantitative analyses. 

Looking first at survey data, Table 8.2 presents the numbers and percentages of principals and teachers 
who reported each of several response options as the reason for their school’s ending its CSR effort. 
School staff cited most frequently losing district “support,” losing funding, losing faculty commitment to 
the reform, and reaching the end of a contract with a CSR model provider as reasons for ending a CSR 
effort. These factors, as well as several options that teachers frequently reported, including lack of 
positive student outcomes, overlap almost completely with the 1 1 factors derived from the literature, the 
sole exception being reaching the end of a contract with a CSR model provider. 

Table 8.2. Reported reasons for dropping a CSR model 



















2002-2003 




2003-2004 




2003-2004 




Reason 


(Principal) 


% 


(Principal) 


% 


(Teacher) 


% 


Lost district support 


20 


28.99 


19 


31.15 


256 


31.84 


Lost funds 


15 


21.74 


15 


24.59 


n/a 


n/a 


Saw no benefit 


6 


8.70 


4 


6.56 


n/a 


n/a 


Incompatible with 
curriculum 


5 


7.25 


1 


1.64 


69 


8.58 


New principal 


1 


1.45 


2 


3.28 


n/a 


n/a 


Contract ended 


n/a 


n/a 


11 


18.03 


116 


14.43 


Lost principal support 


n/a 


n/a 


n/a 


n/a 


119 


14.80 


Lost faculty support 


n/a 


n/a 


n/a 


n/a 


166 


20.65 


Did not improve 
student learning 


n/a 


n/a 


n/a 


n/a 


125 


15.55 


Too difficult to 
implement 


n/a 


n/a 


n/a 


n/a 


34 


4.23 


New reform 


n/a 


n/a 


n/a 


n/a 


118 


14.68 


Other 


13 


18.84 


4 


6.56 


63 


7.84 


Multiple 


9 


13.04 


5 


8.20 






Total usable 
responses 


69 


100.00 


61 


100.00 


804 


100.00 



Of the schools in the NLECSR case study sample, several had dropped their CSR model just prior to or 
during the data collection. Principal and teacher interviews in these schools were reviewed to analyze 
themes related to sustainability and to examine these schools’ specific reasons for ending their reform 
efforts. Loss of funding and loss of district support/backing/priority for CSR emerged as the main reasons 
for dropping among these schools. When asked why her school had discontinued its relationship with its 



One limitation of this analysis is that questions were asked only of those schools that dropped their CSR model. 
Consequently, we cannot assess the extent to which these factors were also present in schools that sustained their 
model affiliations. 
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CSR model and stopped using the program, one respondent made it elear that insufficient funding was at 
the root of her school’s decision: 

I think it had to do a lot with finances. When we got rid of the literacy coordinator and 
the other name I can’t think of, the other one um. . . [Interviewer: The design coach?] 

That’s it. Then they got rid of those positions because we did not have the money, and 
you know they are cutting back, and they are talking about firing teachers and everything. 

I really think a lot of it had to do with money. A lot of it had to do with money. 

A respondent in another school gave similar insight into what was meant by a “lack of district support:” 

My sense is that there’s very little active support on at the district level ... I think schools 
are pretty much left to sink or swim on their own. That’s my impression. And I think, you 
know, the prognosis, given that, is more likely sink. If the district isn’t linked up to it and, 
you know, trying to think about how it can deliver some extra resources, you know, all 
the other things we know have to happen if the place is going to succeed. 

This chapter also seeks to address the extent to which implementation of a particular CSR model persists 
after schools formally drop or switch CSR models. Evidence from case studies suggests that dropping a 
CSR model does not necessarily lead to a dramatic decrease in implementation. At least some schools 
will exhibit sustained implementation of the practices learned during their work with their CSR model 
after they end their affiliation with the model provider. While it appears likely that some CSR practices 
may persist after they have been dropped formally, the question remains as to how strong a determinant 
the relationship with an external CSR model provider is to future levels of implementation. For example, 
when asked about the school’s use of its CSR model, one teacher responded that the model was 
technically inactive but that some residue of the pedagogical approach was still in general use: 

I don’t think we’re active but we still use the tools that we learned from there. Some of 
the programs come and go but then that many of the teachers that go through the training, 
we still implement it even though we’re not quote unquote a [model name] school. 

In another school after the respondent did not list the CSR model among the school’s main improvement 
strategies, she was asked whether there was a reason she did not mention CSR: 

Oh! I’m like, we have so many things . . . the [Model name] model is used. It’s 
disappointing because we don’t have it all so we only have bits and pieces of those that 
came in on the original, [we’re] trying to teach it to the new teachers because the funds 
have run out so we kinda keep some things in place that we had before that don’t require 
much funding. And that’s the study groups where the teachers come and work together. 

So they’re still doing that; that’s still in place. . . . 

I thought it was a wonderful [model], you know, when we had the funding we had all the 
pieces. Right now we’re working with a broken puzzle so we have a little bit of this and a 
little bit of that but it works so much better when you had everything in place. Y ou had 
the extra support, you know, that you could lean on and then if you weren’t quite ready or 
understood it they made you feel really comfortable and they showed you different 
strategies and it was really good. 

Based on the relevant literature and the qualitative analysis of reports of reasons for dropping CSR model 
affiliations (from principals and teachers), we developed a general model of the factors related to 
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sustainability (both sustained reform relationships and sustained implementation of a particular reform 
program). Figure 8.3 depicts this model. 

Figure 8.3. Sustainability logic model 




School-level controls 
Level, size, disadvantage, CSR model 



Improved student achievement 

4. Positive student outcomes 
School NCLB status 
School mean achievement 




District-level variables 

2. Supportive district political context 

3. Sufficient CSR funding 

11. Protection from competing reforms 



Improved social capital 

1 . High local school capacity 
Professional community 

5. Fit or alignment between the reform design 
and the school 

6. Leadership stability 

7. Faculty stability 

8. Faculty commitment including factors 
associated with initial buy-in and the reform 
adoption process 

9. Practical concrete reform specifications that 
are structured into the daily life of the school 

10. Sustained professional development and 
model developer assistance 



Sustained implementation 
Ongoing affiliation with external providers 
Stable or improving fidelity of implementation 




Methods/Analytical Approach 
Data 

The subsample of schools from which we drew data consists of 250 schools that were using a CSR model 
during 2002-03 or 2003-04 and for which we have full implementation data (see chapter 5). Additional 
data were drawn from principal and teacher surveys and principal and teacher interviews. As with the full 
sample, these schools are primarily urban, disadvantaged, low-performing schools. Due to the design of 
the study, we did not restrict our analyses to schools beginning their first year of implementation. 
Twenty-six percent of these schools adopted their CSR model during the first year of the study or earlier. 
Fifty-six percent of the schools adopted their model 3-5 years before the study began. Eighteen percent of 
the schools adopted more than 5 years before the study. Consequently, the majority of schools in these 
analyses had already sustained implementation for a number of years. 

Analytical Models/Methods 

We conducted two main analyses. The first was a multilevel, nonlinear model of the sustainability of 
reform relationships. In this model, we addressed issues concerning dropping or switching CSR affiliations. 
We modeled the likelihood of schools dropping or switching CSR models as a function of district and 
school variables, including the CSR model variable. We hypothesized that measures of 1 1 risk factors 
would be associated with a higher probability of dropping or switching CSR models the following year. 

In this first analysis we fit two multilevel (schools within districts) Bernoulli logistic regression models to 
data on 250 schools that were affiliated with a CSR model in 2001-02, estimating schools’ likelihood of 
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dropping their relationship with their CSR model in either of the next 2 sehool years. The models estimate 
the relationships between one distriet-level variable, several sehool-level variables, and sehools’ 
likelihood of dropping their CSR affiliation. Due to the small number of distriets (21) in our study, only 
one or two distriet-level variables eould be tested in each model. Several other district-level variables 
(e.g., frequent changes in district policy, leadership turnover, concentration of CSR schools, several forms 
of support) were independently tested in similar models to the one presented, with only the one presented 
showing a significant relationship. At the school level, the models each include independent variables that 
control for school characteristics, CSR program, and chronological stage of implementation, and estimate 
the associations between the 1 1 sustainability factors and schools’ likelihood of dropping CSR. 

Let Yij be a binary variable indicating whether school , in district j dropped or switched CSR models in the 
2002-03 or 2003-04 school years. A value of 1 indicates that school , dropped or switched its CSR 
model. The probability model of dropping or switching CSR models can be given as: 

prob{Yij) = 

The nonlinear model for the odds of dropping or switching CSR models is 

n 

ri,j = log[((zJy ) /(I - (p,j )] = y?o / + S l^kj {xSchool variable) ^ 

k=\ 

Where rjij is the odds of dropping or switching CSR models and /? is coefficient of school variables x, for 
all A: = 1 to n. The connection between the odds of dropping or switching CSR models and district 
variables is described in the district level model (level 2). 

Poj = Too + 7 oiiDistrict variable) j + 7 Q 2 iDlstrict variable) ^ 

tP,=trk. 

k=\ k=\ 

The probability can be derived as follows. 

1 

(Pij = ^ 

-(/oo+Toi {District variable) j -^7^2 {District variable) j + ) 

1 + e *■' 



The second model was a multilevel linear model of the sustainability of implementation. This analysis 
measured the change in a school’s implementation from before to after the school dropped its affiliation 
with the CSR model provider. This analysis relied on longitudinal data from the teacher survey. 



The sustainability of implementation analysis fits a three-level hierarchical linear model (teachers within 
schools within districts) using Bernoulli logistic regression to analyze data on the same schools used in 
the preceding analysis (schools that were affiliated with a CSR model in 2002-03). Teacher reports of 
implementation provide the data for the dependent variable. The structure of the model creates a 
school-level slope on the dummy variable indicating the year in which the teacher reported. This slope 
represents the school mean implementation in 2004, adjusted for the prior level of implementation in 
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We estimated a three-level hierarehieal linear model in whieh 

♦ Change in level of school implementation is measured as the mean across component 
implementation indices and 

^ Teacher-level, school-level, and district- level variables predict the observed change. 

ypothesized that if dropping affiliation with a CSR model provider represents a critical event, then 
)ls that dropped their affiliations will experience greater declines in their level of implementation 
those that do not drop their affiliation. 



the level of implementation; 

a set of coefficients to be estimated for teacher control variables; 
i set of time-varying teacher control variables such as grade, subject; 

; the average school level of implementation across both years; 

; a dummy coded 1 if the year is 2003-04. The coefficient on this variable represents the change in 
of school implementation from 2001-02 (prior to ending a reform relationship for all schools 
zed) to 2003-04 (after some schools have dissolved their reform relationship); 

set of coefficients to be estimated for school variables; 

a set of school variables; 

a set of coefficients to be estimates for district variables; 

I set of district variables; 

14 S 

s, district-level error, a school-level error ^ , and a teacher-level error ^ are assumed to be 
ally distributed with a mean of 0. 

isures 

i analysis of the sustainability of reform relationships, the dependent variable was a binary variable 
senting whether the school dropped or switched its CSR model affiliation. It was based on a 
ipal survey item that asked the status of implementation of the model in the previous year. The 
)1 was deemed to have ended their affiliation if the principal responded either that they were no 
;r implementing their model or that they had dropped the prior year’s model and adopted a new 
; 1 . 



;r + ^;r*X + ^o., + + +^. +Ujk 



•e: 



nits of analyses are i teachers (stacked by year) nested in j schools that are nested within k districts; 
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In the sustainability of implementation analysis, the dependent variable was based on the series of 
measures of implementation fidelity described in chapter 5. However, in these analyses, the dependent 
variable was a single general index of implementation that averaged across eight separate components 
(e.g., governance, instruction, family involvement). 

Key independent variables included a series of measures of the presence of the 1 1 sustainability factors 
identified in the literature. When a school had a lack of or low level of a sustainability factor, the school 
was considered to have a risk factor for dropping its reform relationship. For example, a risk factor was 
calculated if there was a lack of leadership stability because a new principal joined the school in 2002-03 
or 2003-04 or if there was a relatively low level of social capital because the school’s score on the teacher 
community scale was below the mean for the sample. Finally, we computed a risk factor index. This risk 
factor index was used to examine the influence of multiple risk factors operating simultaneously as 
opposed to the influence of risk factors operating independently. Although the resulting index of risk 
factors is only a rough estimate of the number of coincident risk factors, it generates a normal distribution 
of 0-9 coincident risk factors and fairly represents the idea, identified in the literature, that multiple 
factors needed to support sustainable reform were absent in a particular school. We also included a series 
of additional covariates at the school level in order to control for other factors that may be related to a 
school’s likelihood of dropping and the level of and change in implementation. 

Three limitations to the study design and data constrain the conclusions we can draw. First, these 
quantitative analyses reveal general relationships, but they do not fully account for the many exigencies 
that may play important roles in sustainability. Continued qualitative work may help us better understand 
this process. Second, the analyses do not differentiate between dropping a reform affiliation and switching 
from one CSR model to another. There may be important differences in when, why, and under what 
circumstances schools switch models as opposed to dropping CSR outright. Lastly, these analyses do not 
address sustained effects on student achievement, only sustained implementation. 

Findings 

Sustainability of Affiliations With CSR Model Providers 

What factors were associated with a higher probability of dropping or switching CSR models? This first 
analysis modeled the likelihood of schools dropping or switching CSR models as a function of district 
and school variables, including the CSR model variable. Table 8.3 presents a schematic of the logistic 
regression. The two sets of columns of results differ only in that the first set treats the sustainability 
factors as independent from each other, and the second set omits the separate factors and, in their place, 
enters the risk factor index, which is a count of the number of coincident risk factors in each school. 

Overall, teacher turnover is significantly related to the likelihood that a school will drop its CSR 
affiliation. The first column of Table 8.3 shows that districts with more serious problems with teacher 
turnover are more likely to have schools that drop their CSR model affiliation. In our models, 
unexpectedly, teacher turnover turned out to be a more important factor than district policy or district 
leadership turnover. This finding highlights the difficulty of sustaining a reform effort when the teachers 
who initially bought into the reform depart and new teachers join the school, necessitating constant 
training and retraining of the fundamental implementers of the reform. 

The school-level estimates indicate that middle schools are less likely to drop their relationship with their 
CSR model developer, and schools with more district-provided professional development activities 
designed to support implementation of CSR are less likely to drop their affiliation. Although schools in 
each of the specific CSR models may have differed from each other prior to participating in the model. 
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Accelerated Schools Project schools were more likely to disengage from that program than schools 
working with “other” CSR models (the omitted category) or SFA. We did not find a consistent 
relationship between years since model adoption and schools’ likelihood of dropping their model. In sum, 
when controlling for the other 1 0 factors, only the factors of teacher retention problems and district 
support through professional development exhibited significant independent associations with the 
likelihood of dropping a reform relationship. 

In the second set of columns, referenced as the cumulative risk factors model, the 1 1 separate variables 
from the prior model have been removed from the model and replaced by a risk factors index. This risk 
factors index attempts to examine whether the key predictor of schools’ dropping a model is the 
cumulative effect of multiple risk factors or the independent effect of each of the factors individually. 

Table 8.3 shows that for each additional risk factor, a school was more likely to drop its affiliation with its 
CSR model provider. Above, there were few indications that risk factors working independently disposed 
a school to drop its model. Here, there is a clear indication that the set of 1 1 risk factors operate in 
combination or cumulatively to dispose schools toward ending their affiliation with their CSR model 
providers. 

Sustainability of Implementation 

Having discussed the sustainability of reform relationships, we will now consider the sustainability of 
reform implementation. That is, do schools continue to implement CSR-like practices over time? Table 
8.4 displays the results from this analysis. The results listed in the “Intercept: implementation in 2002” 
section of the table show whether each factor was significantly associated with the schools’ baseline 
levels of implementation fidelity. The estimates located under the “Slope as outcome” section explore 
which factors are associated with more or less gain in implementation relative to the initial baseline 
measurement. These estimates also address the extent to which dropping affiliation with a CSR model 
provider is associated with a decrease in implementation. The teacher-level covariates at the bottom of the 
table control for changes in the composition of the schools’ faculty over the years and also reveal that 
reading/language arts teachers (compared to mathematics teachers) and teachers with longer tenure at the 
school exhibit higher levels of implementation. 

The key variables of interest were the risk factor index and the indicator variable for schools that dropped 
their CSR model affiliation. The findings suggest that schools suffering from more risk factors began with 
lower implementation levels. Schools with a greater number of risk factors did not improve their 
implementation levels. Schools that formally dropped their models actually began with higher levels of 
implementation than schools that later sustained their relationships with their CSR model providers. 
Schools that dropped their models did also suffer a dip in their implementation levels relative to schools 
that maintained their CSR model affiliation. This implementation dip, however, is a rather modest 5%. In 
other words, schools that dropped their affiliation with their CSR model provider continued to implement 
their models with only 5% less fidelity than when they were formally affiliated with their CSR model 
provider. 

The results show that disadvantaged schools and those that have been identified as low performing and in 
need of improvement gain more implementation fidelity relative to their more advantaged counterparts 
(keeping in mind that nearly all of the CSR schools in our sample are more disadvantaged and lower 
performing than the average school). However, schools with these characteristics began the period under 
study with lower levels of implementation than other schools. There is likely some element of regression 
to the mean and some reason to imagine a ceiling effect because the mean implementation level is fairly 
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high at 73%, with a maximum possible value of 100%. However, sehools are fairly normally distributed 
on implementation, and few approaeh 1 00%. 

Table 8.3. Results from two-level Bernoulli logistic regression estimating the likelihood of 
dropping affiliation with CSR model provider, 2002-03 or 2003-04 



Independent 
Factors Mode 


Cumulative Risk 
Factors Model 


Coeff. 


SE 


Coeff. 


SE 


Intercept 


-0.641 


0.179 


*** 


-.550 


0.797 


kk 


District 

Degree to which teacher turnover is a problem 


0.721 


0.234 


*** 








School 

Has middle grades 


-1.293 


0.423 


•kick 


-1.227 


0.401 


kkk 


School size 


-0.002 


0.174 




-0.027 


0.180 




Disadvantage index 


-0.022 


0.163 




-0.023 


0.155 




Identified for improvement 


0.467 


0.335 




0.551 


0.334 




Accelerated Schools Project 


1.465 


0.679 


kk 


1.616 


0.600 


kkk 


ATLAS Communities 


0.769 


0.798 




1.181 


0.718 




Co-nect 


0.002 


0.692 




0.178 


0.636 




Success for All (SFA) 


-.417 


0.559 




-0.299 


0.511 




Middle stage: 3-5 years since adoption 


-.378 


0.508 




-0.232 


0.514 




Late stage: 5 or more years since adoption 


-.096 


0.412 




0.066 


0.398 




Risk factor index 


- 


- 




.151 


0.090 


k 


Teacher community 


-0.138 


0.256 










Supportive PD for CSR 


-0.326 


0.161 


kk 








CSR grant ended in 2002-03 or 2003-04 


0.503 


0.452 










Change in student reading and mathematics scores 
2000-2001 to 2001-02 


0.163 


0.157 










Applied PD in classroom instruction 


-0.012 


0.156 










Instructional leadership 


0.112 


0.244 










Principal recently joined school 


0.185 


0.307 










Faculty tenure 


0.101 


0.158 










Mandated adoption of CSR 


0.001 


0.326 










Implementation fidelity 2002 


0.003 


0.197 










Usefulness of developer’s assistance 


-0.195 


0.191 










Many competing reform programs 


-0.002 


0.168 











Note: ‘ p < 0.1 0; ** p < 0.05; *“ p < 0.01 . 
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Table 8.4. Results from three-level slope as outcome regression for difference in 
implementation from 2002 to 2004 



Implementation 


Intercept 


0.708 


0.003 


•kick 


District 

Concentration of CSR schools in district 


0.066 


0.003 


** 


Intercept; implementation in 2002 
School 

Has middle grades 


-0.023 


0.006 


kkk 


School size 


0.001 


0.002 




Disadvantage index 


-0.003 


0.002 




Identified for improvement 


-0.007 


0.005 




Accelerated Schools Project 


-0.028 


0.007 


kkk 


ATLAS Communities 


-0.167 


0.010 


kkk 


Co-nect 


-0.001 


0.008 




Other CSR model (SFA is the omitted category) 


0.059 


0.007 


kkk 


Middle stage: 3-5 years since adoption 


0.009 


0.007 




Late stage: 5 or more years since adoption 


0.003 


0.006 




Risk factor index 


-0.006 


0.001 


kkk 


School dropped affiliation with CSR model 


0.006 


0.005 




Slope as outcome: difference in implementation in 2004 adjusting for 
starting level in 2002 slope 


0.034 


0.004 


kkk 


School 

Has middle grades 


0.010 


0.008 




School size 


-0.002 


0.003 




Disadvantage index 


0.003 


0.004 




Identified for improvement 


0.006 


0.007 




Accelerated Schools Project 


0.012 


0.011 




ATLAS Communities 


0.302 


0.014 


kkk 


Co-nect 


0.004 


0.012 




Other CSR Model (SFA is the omitted category) 


-0.014 


0.011 




Middle stage: 3-5 years since adoption 


-0.010 


0.010 




Late stage: 5 or more years since adoption 


-0.007 


0.010 




Risk factor index 


0.004 


0.002 


kk 


School dropped affiliation with CSR model 


-0.012 


0.008 


kkk 


Teacher 

Years of tenure at this school 


0.006 


0.002 


kkk 


Mathematics teacher 


-0.022 


0.003 


kkk 
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Implementation 


Teaches grades 3 and/or 4 


-0.002 


0.004 




Teaches grades 7 and/or 8 


-0.005 


0.006 




Teaches a mix of grades 


-0.002 


0.006 





Note: * p < 0.1 0; ** p < 0.05; *** p < 0.01 . 



There is a similar finding of greater gains having begun at a lower initial level for ATLAS Communities 
and the “other” category of CSR models relative to SFA (which stays fairly stable over time, along with 
Co-nect). It must be noted that the lower initial level is likely due in part to actual differences in fidelity 
and in part to differing difficulties of the specific program keys (which are not controlled for as in 
chapter 5). Further, the different gains between the programs are due to both actual differences and some 
element of a ceiling effect for those programs that started with very high average levels of 
implementation. Schools in the middle or late phases of implementation gain less than schools in the early 
stage (less than 3 years after adoption). 

Conclusion 

One of the greatest challenges, if not the greatest challenge, to CSR is sustaining reform over a period 
long enough to produce substantial effects. By examining how CSRs complete their life course, this 
chapter highlights the importance of studying sustainability as well as the importance of being clear about 
what is being sustained. 



♦ It is critical in future inquiry to distinguish between a sustained reform relationship and sustained 
implementation of a reform. 

♦ Nearly one third of CSR schools in our study ended their relationships with their model 
developers. Of course, that means that the remaining two thirds of schools have sustained a 
reform relationship for more than 3 years, and in some cases more than a decade. 

♦ The results of the sustainability of affiliations analysis indicates that 1 1 risk factors for 
discontinuing a reform relationship operate in combination to dispose schools toward dropping 
their CSR affiliation. 

♦ Resolving faculty retention and providing professional development supports for the CSR effort 
appear to be the most significant of this interrelated set of sustainability factors. These results 
emphasize the critical role of teachers’ knowledge, skills, and commitment as well as faculties’ 
social capital play in sustained reform. In addition, these results situate those factors within the 
practical context of multiple coincident factors that collectively influence schools’ ability to 
sustain reform. 



♦ 



Even with some sense of the factors disposing schools toward discontinuation, it is difficult to 
know what to make of this rate of discontinuation by itself Schools may be dropping their formal 
reform affiliation and also ceasing the implementation of the practices related to that CSR model. 
Alternatively, schools may be discontinuing their reform relationship because they have 
institutionalized the practices of the reform program and have become self-sustaining. Further, 
still other schools may be switching to a new reform program, selecting just a few of the practices 
prescribed by this CSR model to sustain and layer on top the sediment built up from their 
previous history of reform efforts. In sum, an analysis of schools that drop their relationships with 
reform developers is an incomplete analysis of the sustainability of reform. 
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♦ A more complete exploration of sustainability requires the examination of schools’ 

implementation of reform-prescribed practices over several years. Our analyses show that 
although dropping a CSR model affiliation is significantly related to decreases (or less of an 
increase) in implementation fidelity, the magnitude of the effect is fairly modest, a reduction of 
less than 5%. Without a precipitous decline in implementation due to dropping, it is clear that 
many schools that formally drop their affiliation with a reform developer are still sustaining many 
of the practices prescribed by the CSR model developers. 

These analyses do not tell us which of these schools have retained a few fragments of the reform 
practices, which have institutionalized the practices of their CSR model, and which have adapted and 
enhanced the original model in the spirit of continual school improvement, but they do make it clear that, 
on average, the effects of CSR models can persist beyond the formal discontinuation of the reform 
relationship. 
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What Can We Say About the Life Cycle? 

This report began by framing our investigation of CSR within the metaphor of the life eyele. The report 
has traeed the trajeetory of schools implementing CSR, from adoption to implementation to sustainability. 
This chapter seeks to integrate the major findings of each of the preceding chapters. 

We began work on this study at a time of tremendous policy and research interest in CSR. Now codified 
as Part F of Title I of the No Child Left Behind Act (NCLB), CSR remains of substantial policy and 
research salience. Few policymakers or researchers today will openly advocate the notion that school 
improvement will result from ad hoc, piecemeal, or fragmented reform. Almost all agree that school 
improvement will follow only from concentrated and systematic approaches to changing what schools 
and districts do. CSR has been the progenitor of a new focus on entire systems (usually school districts) 
as the entities required to undertake meaningful reform. In an important way, CSR has been scaled up 
from the school building level to the school district level. Thus, the lessons learned from studying CSR’s 
impact and implementation at the school level are even more essential. 

Distilled to their essence, the research questions we examined amounted to three questions: What works 
(to improve student achievement)? How does it work? In what contexts does it work? Below are our short 
answers. 

What Works? 

Success for All (SFA) improves student achievement. While we did not find similarly direct evidence for 
other models, we did find that were they implemented with as much fidelity, ATLAS Communities, 
Accelerated Schools Project, and Co-nect would improve student achievement to a greater extent than 
SFA. For multiple reasons, these three models were not as faithfully implemented as SFA. Our data 
indicate, however, that if they were, those models would appear to behave been even more successful 
than SFA. 

Building social capital, that is, a professional learning community and a sense of collective commitment 
among teachers, also appears to be associated with improved student achievement. Our data show that 
ATLAS Communities and Accelerated Schools Project build the kinds of social capital demonstrated to 
have an impact on student achievement. 

It is worth noting that while our quasi-experimental design allows for causal claims, one should be 
cautious in drawing conclusions not just from the claims made, but from the claims not made as well. 
Other models may work as well, but their effects may have been obscured for a variety of technical 
reasons. 
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How Does It Work? 

Not surprisingly, CSR works when schools faithfully implement the model providers’ design. CSR also 
works when given enough time. We observed no effects until years 3-5 of implementation. It is worth 
reiterating this point about the interplay between implementation and achievement. Even though we see 
substantially similar practices in non-CSR schools, we do not see the same outcomes. The outcomes of 
interest (student achievement and teacher social capital) appear related to the faithful implementation of 
specific models. 

The conventional wisdom on adoption of CSR models (and other reforms as well) suggests that initial 
buy-in on the part of teachers prior to implementation is critical to later implementation. We saw scant 
evidence of such a relationship. Rather, what the data reported here suggest instead is the importance of 
ongoing teacher investment and ownership of the reform throughout implementation, not merely prior to 
launching the reform. 

In What Contexts Does It Work? 

Model developers’ support matters, as do the actions and buy-in of principals and teachers during 
implementation, though not necessarily during the adoption process. We found little evidence of district 
context making much difference. Though the sample included almost two dozen districts, they may still 
be too few to detect any differences across districts. 

So What? 

The specific findings described briefly above and at length in the chapters lead to four lessons for 
practitioners, policymakers, and researchers. 

♦ Institutionalization of CSR 

♦ Ongoing school-level ownership/investment in reform 

♦ Models matter. . . 

♦ ... Up to a point 

Tyack and Cuban (1995) have written persuasively (jointly and independently) about the cycles of 
reform. CSR began to take shape early in the latest cycle following A Nation at Risk. It rose to 
prominence during the middle of the last decade. As such, CSR is no longer new. In many ways, it may 
no longer even be reform per se. Many aspects of CSR have become institutionalized. It is tightly woven 
into the fabric of life of many urban and suburban schools and districts. The warp and woof of school 
reform now includes CSR as a common thread upon which schools pull. We see this in the widespread 
implementation of CSR practices, even among schools that in name at least are not implementing a CSR 
model. As such, buy-in prior to implementation, oft thought of as an important a first step, turns out to be 
less critical than once thought. While necessary, and important, and appropriate, alone it does not lead to 
success, and when missing, it can be overcome. 

Instead, the development and maintenance of investment in and ownership of reform, on an ongoing 
basis, by teachers and principals appears critical to successful, that is, faithful implementation. 

It is faithful implementation of a model that we have found to be associated with positive outcomes, both 
student achievement and social capital (which in turn is linked to student achievement). While we see 
wide institutionalization of CSR, to the point that control schools appear to be “implementing” CSR with 
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comparable fidelity to treatment schools, we see significant and substantial differences in outcomes 
between CSR and non-CSR schools. 

While models matter, the formal relationship between schools and model provider may not matter so 
much; indeed, implementation suffers relatively little when schools drop their contractual relationship 
with their model provider. 

Though few would advocate piecemeal reform, the debate around CSR has shifted subtly but significantly 
since we undertook this study 6 years ago. In 2000, the pressing questions focused on the implementation 
and impact of externally developed CSR models, hence this study and its sister studies and the federal 
grant program to examine these questions. Discussions of CSR now turn as much on the components of 
CSR as they once did on whole models. Six years ago, there was certainly interest in CSR components. 
This study was designed in part to examine the optimal bundle of components. The interest in 
components or individual practices as opposed to entire models, however, has experienced a resurgence in 
recent years (reminiscent of the interest in effective schools of decades past). The fascination with the 
identification of promising practices gives one pause to wonder whether the current reform cycle is 
turning back from a whole-school orientation toward studying effective schools, away from whole-school 
models and towards isolated promising practices. The data from NLECSR suggest caution before 
adopting such an approach. Practices are important, but models matter. The unambiguous conclusion 
from this study is that implementation of CSR-like practices is so widespread that one is tempted to call 
them universal. However, important differences in student achievement growth remain between schools 
that faithfully implement an externally developed CSR model and those that do not explicitly implement a 
CSR model, though they engage in very comparable practices. Practitioners, policymakers, and 
researchers ought to heed this finding before the latest wave of school reform pulls us back to where we 
were before the rise of CSR during the last decade of the 20th century. 
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Table 5.1. A. Regression results for implementation of curriculum 2002 
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Table 5.2.A. Regression results for type of implementation and emphasis upon professional development 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.3.A. Regression results for implementation of inclusion by CSR designation, model, school demographics, and stage 2002 



Appendix C: Additional Tables of Results 









♦ 

♦ 

♦ 


* 

* 

* 


♦ 

♦ 

* 




♦ 

♦ 

♦ 




* 

* 

* 














j 








_ 0) 

(C O *; 

m O 




Z900 


CO 

o 

d 


CO 

o 

d 


in 

o 

d 




o 

CM 

O 

d 


00 

o 

d 


CM 

O 

d 


CM 

O 

d 


o 

CO 

o 

d 




N 

CM 

O 

d 


CO 

o 

o 

d 


o 

o 

d 


o 

o 

d 


O) 

o 

o 

d 


o 

d 


eooo 


x: (0 a 
>- < 




OJ 

OJ 

00 

d 


CM 

CM 

CO 

cp 


05 

CO 

O 

Cp 


CM 

CO 

CO 

d 

1 




00 

CO 

LO 

d 

1 


o 

o 

d 


CO 

N 

o 

d 


CO 

o 

d 


CO 

o 

o 

d 




CO 

CO 

o 

d 


LO 

o 

o 

d 


CO 

o 

o 

d 


o 

o 

<d 


o 

o 

cp 


CO 

o 

o 

d 


CM 

O 

O 

Cp 






■k 

* 


♦ 

* 

♦ 


* 

* 

* 


♦ 

♦ 

* 




* 

♦ 

♦ 




•k 

•k 

k 






















c C 

C o ^ 
(Q .Z O 




OJ 

o 

d 


CO 

o 

d 


CO 

o 

d 


CM 

LO 

O 

d 




o 

CM 

O 

d 


00 

o 

d 


CM 

O 

d 


CM 

O 

d 


o 

CO 

o 

d 




N 

CM 

o 

d 


CO 

o 

o 

d 


Ti- 

o 

o 

d 


o 

o 

d 


O) 

o 

o 

d 


o 

d 


eooo 


§’S| 




00 

CO 

00 

d 


CM 

CO 

(d 


05 

CO 

O 

Cp 


00 

CO 

CO 

cp 




-0.537 


T“ 

o 

o 

d 


CO 

N 

o 

d 


CO 

o 

d 


CM 

o 

o 

d 




CO 

CO 

o 

d 


CO 

o 

o 

d 


LO 

o 

o 

d 


o 

1“ 

o 

<d 


o 

o 

cp 


eooo 


CM 

O 

O 

(d 






* 

* 


♦ 

♦ 

♦ 


* 

* 

* 


♦ 

♦ 

* 




♦ 

♦ 

♦ 




k 

k 

k 






















to 




C\J 

CO 

o 

d 


CO 

o 

d 


CO 

o 

d 


o 

in 

o 

d 




cyj 

o 

d 


00 

T“ 

o 

d 


CM 

O 

d 


CM 

o 

d 


o 

CO 

o 

d 




N 

CM 

O 

d 


CO 

o 

o 

d 


Ti- 

o 

o 

d 


o 

o 

d 


O) 

o 

o 

d 


O 

d 


eooo 


o S 

o O 
O.T 

3 o 
(/) Q. 




05 

N 

N 

d 


CM 

CO 

cp 


o 

N 

o 

cp 


CO 

LO 

CO 

d 

1 




-0.537 


CM 

o 

o 

d 


CO 

N 

o 

d 


CO 

o 

d 


CM 

o 

o 

d 




CO 

CO 

o 

d 


CO 

o 

o 

d 


CO 

o 

o 

d 


o 

<d 


-0.012 


CO 

o 

o 

d 


-0.002 






* 

* 

* 


♦ 

* 

♦ 


* 

* 

* 


♦ 

♦ 

* 




* 

* 

♦ 




k 

k 

k 






















O £ 
t £ <c 




0.033 


CO 

o 

d 


CO 

o 

d 


o 

LO 

o 

d 




(J) 

o 

d 


00 

o 

d 


CM 

O 

d 


CM 

o 

d 


o 

CO 

o 

d 




o 

CO 

o 

d 


CO 

o 

o 

d 


Ti- 

o 

o 

d 


o 

o 

d 


6000 


o 

d 


eooo 


(A 




0.779 


CM 

CO 

cp 


o 

N 

o 

cp 


CO 

LO 

CO 

cp 




-0.537 


CM 

o 

o 

d 


CO 

N 

o 

d 


CO 

o 

d 


CM 

o 

o 

d 




CO 

CO 

o 

d 


CO 

o 

o 

d 


CO 

o 

o 

d 


o 

<d 


CM 

O 

Cp 


eooo 


CM 

O 

O 

Cp 






* 

* 

* 


* 

* 

♦ 


* 

* 

* 


♦ 

♦ 

* 




■k 

■k 




k 

k 

k 














t 








0) 

(0 

(0 

03 o 




'Tt 

CM 

O 

d 


CO 

o 

d 


CO 

o 

d 


o 

LO 

o 

d 




o 

CM 

O 

d 


00 

o 

d 


CM 

O 

d 


CM 

o 

d 


o 

CO 

o 

d 




N 

CM 

O 

d 


CO 

o 

o 

d 


Ti- 

o 

o 

d 


o 

o 

d 


O) 

O 

O 

d 


O 

d 


eooo 


.2 O 

U) 

b 




CO 

00 

d 


CM 

CM 

CO 

cp 


05 

CO 

O 

<D 


o 

CO 

CO 

cp 




00 

CO 

LO 

cp 


o 

o 

d 


CO 

N 

o 

d 


CO 

o 

d 


CO 

o 

o 

d 




CO 

CO 

o 

d 


LO 

o 

o 

d 


CO 

o 

o 

d 


o 

o 

<d 


o 

o 

cp 


eooo 


CM 

O 

O 

Cp 




Intercept 


Accelerated Schools Project Key 


Co-nect Key 


ELOB/TP Key 


MRSh Key 


ATLAS Key 


SFA/KW 


Accelerated Schools Project 


Co-nect 


CL 

o 

_l 

HI 


MRSh 


ATLAS Communities 


School size 


Percentage free lunch 


Percentage non-English speaking 


School has CSR coordinator 


New principal 


Usefulness of developer’s 
information 



111 



American 

INSTITUTES 

FOR Research 



261 






Support for Program Technical 

District Base Comm. Support for Selection Assistance 

Model Outreach PD for CSR Support Support 




AMERICAN 
Institutes 
FOR Research 










Table 5.4.A. Regression results for implementation of influence of assessment 2002 



Appendix C: Additional Tables of Results 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.5.A. Regression results for implementation of informal professional development 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *“ (two-tailed test). 






Table 5.6.A. Regression results for implementation of parent-community outreach 2002 
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Table 5.7.A. Regression results for implementation of pedagogy 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.8.A. Regression results for implementation of instructional time 2002 
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Table 5.9.A. Regression results for implementation of shared decision making 2002 
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Table 5.1 0.A. Regression results for implementation of student grouping 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.11. A. Regression results for implementation of use of technology in classrooms 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.12.A. Regression results for level of total implementation 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.13.A. Regression results for implementation of use of assessments 2002 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 







Table 5.14.A. Regression results for implementation of use of assessments 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.15.A. Regression results for implementation of inclusion 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 







Table 5.16.A. Regression results for implementation of influence of assessments 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 







Table 5.17.A. Regression results for implementation of informal professional development 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.18.A. Regression results for implementation of parent-community engagement 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.19.A. Regression results for implementation of pedagogy 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.20.A. Regression results for implementation of instructional time 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 




Table 5.21 .A. Regression results for implementation of shared decision making 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 






Table 5.22.A. Regression results for implementation of student grouping 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 







Table 5.23.A. Regression results for implementation of use of technology in classrooms 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 










Table 5.24.A. Regression results for total level of implementation 2004 
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Table 5.25.A. Regression results for implementation of type and emphasis of professional development 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 








Table 5.26.A. Regression results for implementation of curriculum 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *** (two-tailed test). 








Table 5.27.A. Change in implementation between 2002 and 2004 
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Significance: p < 0.1 *; p < 0.05 **; p < 0.01 *“ (two-tailed test). 





Table 6.1.A. Descriptive properties of variables used in the social capital analyses 
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Table 6.2.A. Descriptive statistics of the analytical sample used for the analyses of social capital 
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Table 7.1. A. Description of school-level achievement data by state 
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Table 7.2.A. Description of student-level achievement for participating schools in Hickoryville (State C) 
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Note: Eighteen schools (9 pairs) used in the analysis: 9 SFA schools and 8 of them in the sixth year of implementation in 2001-2002. 






Table 7.3.A. Description of student-level achievement for participating schools in Elm County (State A) 
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Table 7.5.A. Description of student-level achievement for participating schools in Rainfield (State N) 
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Table 7.6.A. Description of student-level achievement for participating schools in Riverton (State M) 
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Table 7.7.A. Modeling Multiple Grades Over Year 1 



In the HLM, the weighting process can he done through centering the dummy variables of grades. Below is 
the model description: 

Note: i = student, j = year, and k = school. 

Level 1. Student Level 

'^.jk = Ikjk +Ihjk(Grade_2 )+II^j,(Grade_3 )+ II,j,(Grade_4 )+II,^,(Grade_6 )+II,^,(Grade_l )+ 

^hjk( Gmde_8)+njji,(Gender}hng.i^( LEP)+Tlg.J Black)+^^^J|.(Hispanic)+^^^JJ Olhenninority)+e-ji^ 

Note: Grade 5 as the reference group and grade dummy is grand centered. For the case of Hickoryville, 
grades 7 and 8 will not he included and set grade 4 as the reference group. 

Level 2. Time Level 

//qijj = + Bgjj. (Time) + B^jj, (%free _ lunch) + B^j^ (% nin ority) + B^j (irrplementation year) + 

Bq 5 j (Year * implementation year ) + 

(% minority)+Bo 4 k (implementation year) + 

Note: Time is coded as 2000 = 0, 2001 = 1, 2002 = 2 , and 2003 = 3. So Bqj^. is the mean score for school 
k in 2000 and Bqj^ is average yearly growth from 2000 to 2003. 

Level 3. School Level 

®oo* = + Y(i(^i(CSR) + y^g^i^mplementation) + Yoo^ilniplementation* CSR) + 

YoosiGhange in implementation) + Yoo^iMiddle) + U 

B 3^010 3^01 1 (C SR ) “t Yoi 2 (inip lenient a tion) ~t JJ 

Note: The most interesting parameters in this model are yOl 1 and y012, which indicate impact of CSR on 
the yearly changes in the average achievement score. 

The model descrihed above is relevant particularly for student-level achievement data without link over 
year. In the model proposed above, we will estimate the changes in student achievement at school level 
rather than at grade level. In the Hickoryville case, students are linked over years. The linkage from year to 
year for individual students is no longer needed when multiple grades are included rather than a single 
cohort is followed over years. When employing this approach, we are not taking the advantage of the data, 
the linkage of students over year. 
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Table 7.8.A. Modeling Single Grades Over Year 1 



To model grade -level growth is a simple version as modeling school-level growth. That is, excluding 
the dummy variables of grade. Below is the model description: 

Note: i = student, j = year, and k = school. 

Level 1. Student Level 

^ijk -^hjk Gendei)+Tl 2 ji,( LEP)+Uyi^( Black)+Uyi^(Hispanic)+Tl^jJ Othenninority)+ 



Level 2. Time Level 



//gp = Bggj. “t Bgj j (Time) -t Bgjj {% frcc _ lunch) -I- Bgj^ (% min ority) {implementation year) + 



Bq 5 j {Year * implementation year) -I- 



{%minority)+B04k (implantation year) 



Note: Time is coded 2000 = 0, 2001 = 1, 2002 = 2 , and 2003 = 3. So Bgg^ is the mean score for school k in 
2000 and Bgj^^. is average yearly growth from 2000 to 2003. 



Level 3. Grade (School) Level 

^00* “ J^oco + 5^001 5^002 + Yf^^^i^mplementation* CSR) + 

Ygf 2 s{Change in implementation) + Yooei^i^^dle) + U ^k 



B 3^010 3^01 1 {C SR ) “t You {^mp lementa tion) -r U 

Note: The most interesting parameters in this model are you and yoi 2 , which indicate impact of CSR on the yearly 
changes in the average achievement score. 
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Table 7.9.A. Estimated school-level mathematics and reading Z-score differences between 
CSR and matched comparison schools 2000-2001 to 2003-2004 





Model for Difference in 
Math Z-Score 


Model for Difference in 
Reading Z-Score 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


-0.05 


0.062 


-0.09 


0.069 


Year indicator: 2001 


0.04 


0.038 


0.02 


0.035 


Year indicator: 2002 


0.05 


0.038 


-0.07* 


0.035 


Year indicator: 2003 


0.06 


0.038 


0.00 


0.035 


Z-score difference in 2000 


0.64*** 


0.095 


0.67*** 


0.094 


Z-score average in 2000 


-0.12 


0.085 


-0.07 


0.098 


Average percentage of students who are eligible for free or 
reduced-price lunch 


-0.07 


0.071 


-0.05 


0.080 


Average percentage of minority students 


-0.09 


0.073 


0.06 


0.082 


Average school size (unit per 1 ,000 students) 


-0.13 


0.081 


-0.09 


0.090 


Average student-teacher ratio (unit of 10 students per 
teacher) 


0.05 


0.077 


0.06 


0.086 


Average difference in percentage of students who are 
eligible for free or reduced-price lunch 


-0.31 


0.368 


-0.11 


0.419 


Average difference in percentage of minority students 


-1.01** 


0.394 


-0.42 


0.434 


Average difference in school size (unit per 1 ,000 students) 


-0.02 


0.019 


-0.01 


0.021 


Average difference in student-teacher ratio (unit of 10 
students per teacher) 


0.01 


0.022 


0.04* 


0.024 


Estimates 










Combined effect in 2001, 2002, and 2003 


0.16* 


0.093 


-0.05 


0.086 


Variance 










School level 


0.31*** 


0.060 


0.43*** 


0.074 


Time level 


0.45*** 


0.034 


0.37*** 


0.029 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Table 7.1 0.A. Estimated Z-score differences in mathematics and reading between CSR and 
matched comparison schools, by implementation year 2000-2003 



Model for Math 


Model for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


-0.181 


0.115 


-0.223* 


0.118 


Year indicator: 2001 


0.012 


0.121 


0.034 


0.112 


Year indicator: 2002 


0.083 


0.121 


-0.061 


0.112 


Year indicator: 2003 


0.149 


0.121 


0.110 


0.112 


Middle age indicator (3-5 years of implementation) 


0.081 


0.172 


0.390** 


0.177 


Old age indicator (more than 5 years of 
implementation) 


0.027 


0.325 


0.008 


0.334 


Z-score difference 1999 


0.577*** 


0.079 


0.626*** 


0.081 


Average Z-score 1 999 


-0.081 


0.067 


-0.044 


0.075 


Interaction: middle age and year 2001 


0.258 


0.184 


0.025 


0.170 


Interaction: middle age and year 2002 


0.189 


0.184 


-0.190 


0.170 


Interaction: middle age and year 2003 


-0.047 


0.184 


-0.289* 


0.170 


Interaction: old age and year 2001 


-0.264 


0.336 


-0.008 


0.309 


Interaction: old age and year 2002 


-0.436 


0.336 


-0.177 


0.309 


Interaction: old age and year 2003 


0.126 


0.336 


0.061 


0.309 


Average percentage of students who are eligible for 
free or reduced-price lunch 


-0.062 


0.074 


-0.053 


0.081 


Average percentage of minority students 


-0.077 


0.073 


0.076 


0.082 


Average school size (unit per 1 ,00 students) 


-0.125 


0.081 


-0.088 


0.089 


Average student-teacher ratio (unit of 10 students 
per teacher) 


0.048 


0.077 


0.060 


0.086 


Average difference in percentage of students who 
are eligible for free or reduced-price lunch 


-0.325 


0.368 


-0.142 


0.416 


Average difference in percentage of minority 
students 


-1 .029** 


0.394 


-0.431 


0.432 


Average difference in school size (unit per 1 ,000 
students) 


-0.015 


0.019 


-0.010 


0.021 


Average difference in student-teacher ratio (unit of 
10 students per teacher) 


0.054 


0.218 


0.379 


0.241 


Variance 


School level 


0.311*** 


0.060 


0.420*** 


0.073 


Time level 


0.441*** 


0.034 


0.374*** 


0.029 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Appendix C: Additional Tables of Results 



Table 7.11. A. Estimated improvement in student-level mathematics and reading scores for 
CSR and matched comparison schools in Hickoryville (State C) 2000-2001 to 2003-2004 





Model for Math 


Model for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


656.9*** 


41.0 


903.5*** 


46.0 


CSR 


22.8*** 


1.1 


21 .9*** 


0.9 


Time 


-2.6 


7.3 


6.9 


10.4 


CSR*Time 


-0.7 


1.6 


-0.5 


1.3 


Girl 


3.6*** 


1.1 


8.7*** 


1.2 


ESL 


-11.8*** 


1.6 


-1 4.2*** 


1.6 


Free lunch 


-3.7** 


1.6 


* 

CO 


1.5 


Black 


-14.7*** 


3.5 


-15.2*** 


3.5 


Hispanic 


-3.0 


3.5 


-6.2* 


3.5 


Other 


-6.1 


6.4 


-6.5 


6.4 


Average percentage of students who are 
eligible for free or reduced-price lunch 


24.2 


18.1 


-67.1*** 


17.6 


Average percentage of minority students 


-96.1** 


38.7 


-274.3*** 


43.6 


Average school size (unit per 1 ,00 
students) 


-1.1 


3.5 


6.3* 


3.7 












Variance 










Intercept, school level 


214.4** 


473.4 


473.4** 


214.4 


Covariance, school level 


-32.4** 


-40.8 


-40.8** 


20.3 


Time, school level 


9.9** 


6.6 


6.6*** 


2.8 


Intercept, student level 


1071.3*** 


1138.6 


1138.6*** 


40.2 


Covariance, student-level 


-38.1*** 


-53.6 


-53.6*** 


10.1 


Time, student level 


7.0** 


14.6 


14.6*** 


3.8 


Time level 


367.7*** 


322.9 


322.9*** 


8.2 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 




AMHRICAN 
INSTITUTES 
FOR Rf:skarc;h 



331 






Appendix C: Additional Tables of Results 



Table 7.12.A. Estimated improvement in student-level mathematics and reading scores 
for CSR and matched comparison schools in Elm County (State A) 2001-2002 to 2003-2004 





Model for Math 


Model for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


4532.1** 


1568.600 


1721.8 


958.030 


CSR 


3.8 


4.494 


8.2** 


3.412 


Time 


-18.2 


10.709 


-13.9 


8.373 


CSR*Time 


-0.1 


6.289 


1 1 .0** 


4.720 


Girl 


4.5** 


2.213 


1 1 .7*** 


1.616 


ESL 


-15.8*** 


3.641 


-20.8*** 


2.771 


Free lunch 


0.2 


4.305 


-5.5 


3.818 


Black 


-3.8 


21.322 


-15.8 


16.043 


Hispanic 


34.2 


21 .245 


12.9 


15.988 


Other 


35.4 


23.957 


8.0 


18.548 


Average percentage of students who are 
eligible for free or reduced-price lunch 


-10.5 


150.380 


-292.3** 


90.805 


Average percentage of minority students 


-4547.2** 


1708.880 


-1202.1 


1042.980 


Average school size (unit per 1 ,000 
students) 


-0.9 


0.571 


0.0 


0.348 


Average student-teacher ratio (unit of 
10 students per teacher) 


16.3** 


4.636 


2.3 


2.859 


Variance 










Intercept, school level 


70.7 


70.473 


79.8 


59.838 


Covariance, school level 


-23.8 


45.614 


-53.0 


37.240 


TIME, school level 


93.0 


48.994 


48.1* 


28.130 


Intercept, student level 


2974.2*** 


124.370 


3625.3*** 


139.970 


Covariance, student level 


-185.1*** 


48.665 


-1253.8*** 


67.060 


TIME, student level 


113.2*** 


32.754 


524.5*** 


42.944 


Time level 


1189.5*** 


39.784 


1090.8*** 


37.123 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Appendix C: Additional Tables of Results 



Table 7.13.A. Estimated improvement in student-level mathematics and reading scores 
for CSR and matched comparison schools in Riverton (State M) 2001-2002 to 2003-2004 





Model for Math 


Model for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


625.2*** 


16.202 


629.7*** 


15.749 


CSR 


-3.5 


10.367 


-1.3 


8.733 


Time 


18.7*** 


3.284 


16.7*** 


2.236 


CSR*Time 


2.0 


4.624 


1.1 


3.138 


Girl 


-10.6*** 


3.676 


-1 1 .5*** 


3.772 


ESL 


-5.1 


4.487 


-7.9* 


4.642 


Special education 


1.8 


7.620 


12.5 


7.902 


Free lunch 


8.7*** 


2.038 


15.8*** 


2.108 


Black 


-5.7 


5.853 


2"^ 2*** 


6.062 


Hispanic 


-1.9 


2.731 


-3.0 


2.773 


Other 


0 2*** 


1.852 


6.8*** 


1.901 


Average percentage of students who are 
eligible for free or reduced-price lunch 


-43.5* 


19.044 


-36.8 


19.041 


Variance 










Intercept, school level 


200.2 


126.260 


136.5 


91.139 


Covariance, school level 


-65.8 


47.533 


-27.6 


25.318 


TIME, school level 


36.0 


23.612 


13.5 


10.432 


Intercept, student level 


443.0*** 


64.345 


518.8*** 


72.690 


Covariance, student level 


176.7*** 


36.021 


178.3*** 


37.073 


TIME, student level 


63.8* 


33.497 


12.0 


30.904 


Time level 


681.5*** 


39.763 


741 .4*** 


42.024 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Appendix C: Additional Tables of Results 



Table 7.14.A. Estimated improvement in student-level mathematics and reading scores for 
CSR and matched comparison schools in Dodgeland (State D) 1999-2000 to 2003-2004 





Model for Math 


Model for Reading 


Effect 


Estimate 


SEr 


Estimate 


SE 


Intercept 


157.7*** 


8.0 


160.5*** 


6.7 


Dummy: 2000-2001 


-| _4*** 


0.5 


-1 .6*** 


0.5 


Dummy: 2001-2002 


2 Q*** 


0.4 


-0.3 


0.4 


Dummy: 2002-2003 


3.5*** 


0.4 


-0.4 


0.4 


Dummy: 2003-2004 


6.9*** 


0.5 


-j Y*** 


0.5 


Dummy centered: grade 5 


0.4*** 


0.1 


0.3*** 


0.1 


Dummy: CSR 


-3.0* 


1.5 


-2.6* 


1.3 


Interaction: Y2001*CSR 


-0.4 


0.6 


0.3 


0.6 


Interaction: Y2002*CSR 


1 .8*** 


0.6 


0.2 


0.6 


Interaction: Y2003*CSR 


1 .6*** 


0.6 


-0.3 


0.6 


Interaction: Y2004*CSR 


0.7 


0.6 


-0.2 


0.6 


Average percentage of students who are eligible for free 
or reduced-price lunch 


22.4** 


9.6 


7.9 


8.1 


Average percentage of minority students 


-37.1*** 


7.8 


-23.7*** 


6.6 


Average school size (unit per 1 ,000 students) 


0.1 


0.2 


0.2 


0.2 


Average student-teacher ratio (unit of 10 students 
per teacher) 


0.4 


0.3 


0.2 


0.2 


Variance 










School level 


10.1*** 


3.4 


y >1 *** 


2.4 


Time level 


146.0*** 


1.6 


142.7*** 


1.6 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Appendix C: Additional Tables of Results 



Table 7.15.A. Estimated improvement in student-level mathematics and reading scores 
for CSR and matched comparison schools in Rainfield (State N) 1999-2000 to 2003-2004 



Model for Math 


Modei for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


83.7*** 


11.3 


85.7*** 


15.1 


Dummy: 2000-2001 


5.4*** 


0.5 


7.9*** 


0.6 


Dummy: 2001-2002 


8.0*** 


0.6 


9.5*** 


0.7 


Dummy centered: grade 4 


-0.5 


0.4 


3.4*** 


0.4 


Dummy centered: grade 5 


2 "Y*** 


0.4 


2 0*** 


0.4 


Dummy: CSR 


-2.9** 


1.4 


-2.9 


1.9 


Interaction: Y2000*CSR 


2 0**** 


0.7 


3.1*** 


0.8 


Interaction: Y2001*CSR 


1.4 


0.7 


2 /| *** 


0.9 


Average percentage of students who are eligible for free or 
reduced-price lunch 


-23.8 


16.1 


-21.2 


21.4 


Average percentage of minority students 


9.1 


21.4 


0.9 


28.5 


Average school size (unit per 1 ,000 students) 


0.3 


0.4 


0.4 


0.5 


Average student-teacher ratio (unit of 10 students per 
teacher) 


0.0 


0.4 


0.0 


0.5 


Variance 




1 


School level 


15.4*** 


4.2 


27.5*** 


7.4 


Time level 


364.9*** 


3.9 


560.9*** 


5.9 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 






Appendix C: Additional Tables of Results 



Table 7.16.A. The 2002 implementation score of PD type and difference in mathematics and 
reading Z-Score between CSR schools made from comparison schools 2000-2003 





Model for Math 


Model for Reading 


Effect 


Estimate 


SE 


Estimate 


SE 


Intercept 


-0.317 


0.505 


0.104 


0.579 


Year of implementation 


0.037 


0.025 


-0.003 


0.027 


Average score of 2002 implementation: PD type 


0.112 


0.656 


-0.236 


0.754 


Difference in 2002 implementation score: PD type 


0.587 


0.470 


1.059* 


0.541 


Z-score difference 1999 


0.652*** 


0.082 


0.673*** 


0.094 


Average z-score 1 999 


-0.028 


0.077 


0.091 


0.088 


Average percentage of students who are eligible for 
free or reduced-price lunch 


-0.129* 


0.077 


-0.089 


0.088 


Average percentage of minority students 


-0.058 


0.073 


0.036 


0.084 


Average school size (unit per 1 ,000 students) 


-0.074 


0.078 


-0.050 


0.090 


Average student-teacher ratio (unit of 10 students 
per teacher) 


0.059 


0.077 


0.112 


0.088 


Average difference in percentage of students who 
are eligible for free or reduced-price lunch 


-0.261 


0.381 


-0.634 


0.437 


Average difference in percentage of minority 
students 


-0.752* 


0.418 


0.096 


0.481 


Average difference in school size (unit per 1 ,000 
students) 


-0.004 


0.019 


0.002 


0.021 


Variance 










Intercept, school level 


0.460** 


0.184 


0.791*** 


0.233 


Covariance 


-0.063* 


0.041 


-0.115** 


0.050 


Year of implementation 


0.018*** 


0.009 


0.031*** 


0.011 


Time level 


0.383*** 


0.034 


0.326*** 


0.029 



Note: * p value significant at 0.1 level; ** p value significant at 0.05 level; *** p value significant at 0.01 level. 
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Appendix C: Additional Tables of Results 



Table 8.1. A. Reported reasons for dropping a CSR model 

















Reason 


2003 

(Principal) 


% 


2004 

(Principal) 


% 


2004 

(Teacher) 


% 


Lost district support 


20 


28.99 


19 


31.15 


256 


31.84 


Lost funds 


15 


21.74 


15 


24.59 


n/a 


n/a 


Saw no benefit 


6 


8.70 


4 


6.56 


n/a 


n/a 


Incompatible with 
curriculum 


5 


7.25 


1 


1.64 


69 


8.58 


New principal 


1 


1.45 


2 


3.28 


n/a 


n/a 


Contract ended 


n/a 


n/a 


11 


18.03 


116 


14.43 


Lost principal support 


n/a 


n/a 


n/a 


n/a 


119 


14.80 


Lost faculty support 


n/a 


n/a 


n/a 


n/a 


166 


20.65 


Did not improve 
student learning 


n/a 


n/a 


n/a 


n/a 


125 


15.55 


Too difficult to 
implement 


n/a 


n/a 


n/a 


n/a 


34 


4.23 


New reform 


n/a 


n/a 


n/a 


n/a 


118 


14.68 


Other 


13 


18.84 


4 


6.56 


63 


7.84 


Multiple 


9 


13.04 


5 


8.20 






Total usable 
responses 


69 


100.00 


61 


100.00 


804 


100.00 
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